21

Sanity metrics over vanity metrics

In today’s fast-changing business world, measuring impact in real time through agile gives us the benefit of real-time reporting and instant feedback. The purpose of metrics is to inform us whether what we are doing is working. Metrics track success and performance against goals, objectives and targets to establish if we are working effectively.

Metrics should tell us what we are doing well, show us where we can do better and help steer us in the right direction. Good metrics can tell us two key things about the work we are delivering:

1. Are we doing the right thing? Measures of success tell us if our work satisfies and meets the needs of its users by delivering the value and benefits needed. These metrics tell us we are on track with our solutions and they are proving to be effective and successful, or not.
2. Are we doing the thing right? Performance metrics give us insight into whether the approach we are taking is an efficient and effective approach. These metrics establish whether our performance results in a return on investment: that more is gained than is spent. These metrics help to justify that the results are worth the work and effort needed to achieve them.

Vanity metrics

Vanity metrics are the sort of metrics that tell you only what you want to hear and that look good: they massage our egos and paint a picture of the best parts of the story rather than the whole story. Vanity metrics can be misleading, as often they tell only part of the story and show the positives, which have little value and act as a distraction from what may be happening beneath the surface.

Choosing the right metrics and measures of performance can be the difference between success and failure. Metrics should be informative and reveal opportunities to improve and expose problems and risks to our performance. As well as showing where we are doing well, it should flag areas we can do better.

‘It would be nice if all of the data which sociologists require could be enumerated because then we could run them through IBM machines and draw charts as the economists do. However, not everything that can be counted counts, and not everything that counts can be counted.’

CAMERON, 1963

Sales vanity metric

This sales vanity metric (Figure 21.1) reports a good volume of leads and a great conversion to prospects, which would suggest that this sales pipeline has good prospects in terms of generating sales with lots of validated potential customers in the pipeline. This metric does tell us that we have a good volume of prospects, but it does not tell a complete story.

If we look at the next phases of the pipeline (see Figure 21.2) we can see that there are only three sales completed and no repeat sales. So, the conversion rate for actual sales is 1 per cent. If the item being sold is a repeatable purchase rather than a one-off, then the lack of repeat sales would be a red flag to the sales team.

The initial metric might work to keep management happy in the short term but if no sales are generated and sales are not repeating, ultimately the sales method in practice is not sustainable.

Figure 21.1 Vanity metric

Figure 21.2 Sanity metric

If the metrics were to show the entire pipeline, they would show the success of a large volume of qualified prospects, but that those prospects are not converting to paying customers and, perhaps more so, that some of them are not choosing to buy again. Using this information, the team can revisit the activities across the pipeline to establish if there is an issue with the selling process, or if the prospects simply are not the right kind of customer and there is an issue with the validation of a lead to a prospect.

Sanity metrics

Sanity metrics are the measures that tell us what we need to know in order to leverage success and address weaknesses to ensure that we are both delivering the right thing, and delivering the thing right. Sanity metrics help us to identify how much work we are achieving, and what value that work delivers.

Sanity metrics tell us what is actually happening and how we are progressing against our goals and objectives. These metrics are real-time, true representations of the current state of play and performance trends that inform decision making and drive direction for future work.

Metrics should provide ratios and rates that tell us continuously where we are and measure if our actions are having positive or negative effects on the work we are delivering.

Using the sales example, if we are looking to increase sales, a more suitable sanity metric would be to create a ratio of prospects to sales. This would provide us with a conversion rate that can be established as a metric upon which we wish to improve.

The conversion rate can be monitored, as a variety of ideas are tested to increase the conversion rate. A second action would be to test the validation of prospects as genuine potential buyers, in order to ensure there is a product–customer match and that there is demand for the product.

Another example of a good sanity metric is a repeat or retention rate. Many businesses are reliant on customers being retained, as the cost of an initial sale is generally more expensive than repeat sales or intervention because the customer is easier to reach, whereas new customers require prospecting and time to convert.

Social media businesses are a great example of where the repeat metric is vital. It is not the number of sign-ups to the network that is the key metric, it is the number of times that a user revisits and engages on that network that creates the value. Maintaining engagement and retaining customers is key to most business models, and metrics should help to measure this and report if any anomalies appear so that they can be analysed and acted upon quickly.

Measures of success

A key value metric is one that enables us to measure the value of our work against our objectives, which may be to improve our work, making it better, faster or cheaper than before. Results are compared with the acceptance criteria to establish whether our work is fit for purpose, satisfies the requirements and produces expected benefits.

When we measure the value of our work, we want to establish whether we are achieving the desired results. Not all these measures work in harmony with one another and so it is important to clarify what it is being measured against. There is a need to balance our work: if a key objective is to improve quality, then the volume of work we are capable of achieving may drop, as a higher quality of work generally requires more time and resources to achieve.

Releasing work early, using tactics such as an MVP and test-driven development, provides us with metrics that can be measured against our estimations and rankings to establish if we are on target with our solutions, by analysing value metrics through using feedback-driven development.

Value metrics

Review of value can be achieved from gathering feedback on levels of satisfaction, levels of engagement and return on investment. These metrics are not necessarily numbers, and not all value is created at the point of creation or delivery; sometimes value comes later.

An example of indirectly measurable value, the goal of increasing visibility of a product or service to increase sales is a popular marketing activity. Here the goal is to raise awareness of a product or service and, in turn, this creates more prospects to convert to a sale. The value and the benefit are delivered in terms of prospects bringing more potential sales rather than actual sales.

Added to this is the quality dynamic: while the activity has led to raised awareness, the value, in terms of sales, may not change if the people who are now aware of your product are not interested in buying it, conversion rates decrease and other performance metrics report a negative impact. Working this out as early as possible is vital to enable change.

Performance metrics

There are a number of metrics we can use to measure performance against our targets and goals that enable us to see whether we are delivering work that is of the right quality, at the right time, to provide gain and benefits that move us towards reaching our goals and targets.

Performance metrics identify if the value that the work is producing is worth the effort being made. To be sustainable, we must create more value than we are consuming in our efforts and so deliver a positive return on investment.

Metrics also can help to identify waste and opportunities for improving current processes and systems to deliver work more efficiently and effectively.

Volume metrics

A volume metric is the amount of work being achieved and completed, and can be compared to what the initial forecast showed could be done. Identifying if we are under- or over estimating our workloads can help to ensure we schedule a workload that is achievable without the need to cut corners or constantly miss targets.

By measuring the amount of work we are achieving in our sprints, we can start to measure our velocity towards our goal. By recording actuals, as well as our estimations of activities, we can calculate at the end of the sprint how much work has been achieved and gain insight into how we may have to refactor our previous estimates for future work.

As well as reviewing the work that has been completed, it is valuable to look at work that has not been done. A tactic within agile is to reduce the amount of work required by identifying work that does not need to be done in order to reach the goal. This may be taking our wasted time or activity, or features that are expensive to create and deliver low value, or are rarely required.

On average, when using software, we use only 20 per cent of features to carry out 80 per cent of activities – this statistic has led to the development of new software and devices that focus on delivering a limited set of features that still provide a viable solution most of the time. A good example of this is tablets versus laptops: the functionality of a tablet is focused on email, internet and social media, which is far more limited than a laptop. However, because these features make up a large proportion of laptop use, the tablet has become a viable alternative, providing easier and simpler accessibility to these services.

Work not done

Simplicity – the art of maximising the amount of work not done – is essential.

One of the key principles behind agility is keeping things simple. Agile works on the principle that solutions should be built to be fit for purpose. Often, there is a degree of work included in a scope that does not deliver significant value, or is above and beyond the requirements of a solution, and so may not need to be completed straight away or even at all.

If we can establish better, faster, more efficient ways of working, then we can reduce the amount of work needed, and so there will be an amount of work that is no longer required to be completed. Projects often are over-scoped and so, very likely, there is work in the backlog that will never be completed and will be discarded or stored away for future action, should it be needed.

Velocity

Agile enables change with the addition and removal of tasks within the work yet to be done. When we have a backlog of work, and an average rate at which we are completing work, we can map how our work is reducing or increasing over time. With this information, we can track our velocity and predict when we are likely to have completed a volume of work.

Burn up/down charts

Burn up/down charts are popular in software development for showing progress over time and the backlog of work. In this graph (Figure 21.3), a project with 500 points of work has been defined, and the team estimate they can achieve 50 points of work per week. Based on this estimation, the time to complete activities will be 10 weeks. The light grey line shows this average.

The dark grey line shows actuals: in weeks 1 and 2, progress is slow and only 30 points of work are achieved each week; however, in week 3, the chart decreases by 150. There are a couple of reasons for this decrease: the team achieved their 50 points of work, but also, during the sprint review, 10 points of work were removed from the backlog, as they had become unnecessary.

In week 4, the team achieved 50 points of work and were on the path to complete early; in week 5, 100 points of work were achieved, as extra resources had been made available. These extra resources highlighted some issues during the review meeting and a number of unforeseen activities were added, which put the team back behind target. Over the course of the next four weeks, an average amount of work was achieved and some last few activities were discarded as the project came to an end.

Plotting the volume of work onto a graph, we can see the rate of burn down or burn up of work. By visualising the work in this way, we can see clearly the impact of adding and removing work from the workload. In some cases work may continue to escalate. If a burn chart is being produced regularly, a trend should begin to appear which will flag that time, and resources will not be sufficient to fulfil the volume of future work in the backlog.

Figure 21.3 Burn up/down chart

These charts enable you to see the velocity of work in progress and measure this against where it was predicted to be. It can help you to report back the impact of change to the scope of work and the effect that increased, or ad hoc, work has on workflow.

Measuring change

Understanding progress through metrics and visualising the movement of work over time can allow us to pre-empt change and enable us to adapt our practices, not only to improve but also to sustain our current positions. If we do not respond to our changing environment, we can lose traction and momentum.

Change can come from internal decisions or external changes in the environment in which we operate, and both can impact upon our ability to deliver value and sustain performance. By measuring value and performance regularly, it is possible to identify and track trends and changes to the amount of work or the quality of work over time. This can help inform future decision making and identify opportunities for improvement.

If there is a target to scale work and increase throughput, metrics can be key indicators that show where existing processes and systems may struggle with the increased workload and are hitting capacity, or where the increase may be causing a bottleneck or a decline in performance.

One key benefit of digital agile tools is that they can calculate performance metrics automatically and display them in the form of graphics, such as burn down charts, rather than being manually produced. This is a key benefit of digital tools, but has to be weighed against the benefits of a physical board and doing the calculations manually. As with maths and calculators, while the calculator improves efficiency, it does remove the benefit of the learning that comes from working it out manually and being able to see the workings, which can be of great benefit to increase understanding of the metrics and the information they are providing to us.

One solution is the use of touch-screen technology to display boards, which provides a visual and still tactile dashboard while providing the computing power to quickly and efficiently create and display real-time performance metrics.