INTRODUCTION

This is a book about user acceptance testing (UAT) in its many forms and the uses to which it is put. It draws together many strands of material about testing, project management, quality management, team behaviour and other relevant pieces of the complete UAT experience and weaves them into a strong and reliable lifeline for the novice UA tester or stakeholder.

The book has been written to meet the needs of three disparate groups of people. The first of these groups is those who are directly involved in the UAT exercise. For this group we aim to provide a practical and fairly complete guide to the testing of information systems that contain software. As the subtitle indicates, we have adopted a step-by-step approach to enable them to acquire the necessary terminology (jargon) and basic principles as they learn about the practical challenges of UAT and how to deal with them. Within this group we include not only those asked to do the testing (usually end-users of the system) but also those who will have commissioned the system and the testing (sponsors) and those who will be expected to deliver the expected benefits (business managers). The book addresses this group as a whole, but also identifies the specific challenges and provides a practical guide for each of the subgroups within it.

The second group is the professional testers or developers who have been asked to support UAT, or who may simply wish to better understand why UAT is both important and challenging. For this group we explain what kind of support may be needed and why, and we explain where UAT fits within the overall context of structured testing and development life cycles.

The third group is made up of those professionals for whom UAT is an essential ‘tool of the trade’; project managers, quality managers and test managers for example. For this group we seek to place UAT within the overall disciplines of testing, quality management and project management.

Above all the book is intended to explain and explore the significance of an exercise that is commonly neglected in books about testing and often overlooked in books about project management and quality management, yet which is a bridge between these disciplines and, in some respects, is an essential practical expression of each of them.

WHAT THIS BOOK IS ABOUT

Information systems (ISs)

Generally speaking we acquire software to enable us to achieve something of value to us, such as playing a game, making our first million on the stock exchange, or making our business run more efficiently. In some cases software can achieve its purpose without human involvement (except to press a button), but in most cases software interacts with people in a way that achieves a desired result. For example an air traffic control system might provide information to air traffic controllers about where aircraft are, where they are going, how fast and so on. The air traffic controller makes decisions about where each aircraft should go to ensure they are all safely separated and the system relays this information to the pilots of the aircraft. So the purpose of the overall system is to keep aircraft safe and to do that it needs suitably trained people, the air traffic control software, some hardware and some organisation (for example to provide communications between air traffic controllers and pilots). Although this is quite a complex example, it has all the characteristics of an IS. What it most importantly demonstrates is that software is not something that operates in isolation. What users are interested in is not just whether the software does what it should do but whether the system as a whole achieves its purpose and whether they, as users, will be able to operate the system effectively (and without undue stress).

Figure 0.1 An information system

In a business context we would have business processes that involve human beings using information to add business value, such as a supermarket checkout assistant adding up all the retail values of the items purchased by a customer, preparing a bill for the customer to pay, accepting payment and providing a receipt. All of this could be done manually, but this is a process that is usually automated for reasons of efficiency and for the advantage of having all the details of each transaction available as information to be processed in making other decisions such as when to restock. In this example the effectiveness of the overall process would probably be measured by how quickly and accurately a single customer’s shopping can be handled. Here again the whole process is measured, not just the part the software does, because that is the process that adds value to the business. From a user perspective a checkout operator will be sitting at a checkout for perhaps hours at a stretch. To them it will be vitally important that the efficient operation of the checkout is achievable routinely and without heroic effort. If the scanner refused to scan some items, the conveyor broke down at regular intervals or the bills produced were even occasionally incorrect, the impact on the user would undoubtedly be increased stress.

So this book is about testing ISs and not just software, and it is also about considering all the perspectives of all the people involved in building, operating and using the system to achieve its expected business benefits.

Testing ISs

When we set up a UAT exercise we are testing a system and not just a piece of software, which means we are really interested in knowing whether the overall IS works. If the system does not work as we expect, there could be a number of reasons, and among these is the possibility that the software does not do what it should. But the software could be working perfectly and some other aspect of the system could be at fault.

To make this as clear as possible we need to take a brief look at how ISs are created – not in detail but as a high-level view of what happens to an IS from the first idea through to the realisation. This high-level perspective is usually called a life cycle.

The life cycle of an IS is simply a model of everything that happens to it from the time it is first envisaged to the time it is retired. Figure 0.2 shows a simple schematic view of what an IS life cycle might look like.

What this life-cycle model describes is a process that begins with requirements (which are the means of describing what the system needs to do), proceeds through a vague and shadowy phase called development (which we will only delve into occasionally and only if we absolutely need to understand something about it) before reaching UAT as the final stage before the system is ‘released’ into service. We will take a closer look at how requirements are arrived at and documented in Chapter 2.

Figure 0.2 Life cycle of an IS

One key feature of this life cycle is the balance of time, effort and money spent on it. The part of the life cycle that usually gets most of the attention is the initial development, even though that accounts for only about 20 per cent of the total spend. Most of a system’s life is spent in being used, maintained, modified and repaired. The better the system is at the end of development, the lower that large chunk of life-cycle cost is likely to be – and the ‘gatekeeper’ between development and the rest of the life cycle is UAT. Effective UAT can avoid excessive maintenance costs (for example repairing defects), reduce the cost of modifications (for example to improve the system’s usability) and prolong the life of the system, so giving a better return on the original investment in development.

UAT is the final frontier between initial development and the rest of the system’s life, a frontier beyond which could be a world in which your new IT system makes your life easier, anticipates your needs, saves you time and always turns out great results – or a world where using your IT system is a nightmare in which nothing functions, the system seems determined to prevent you completing your work and the outputs are never what you wanted.

Anyone accustomed to using IT systems knows that these two ‘parallel universes’ are actually very close together and that predicting which of those worlds you will be transported to when your system is declared operational is not easy. That is the role of UAT – to provide a glimpse of the future (just) in time to avert a disaster if that is what you foresee and to give the users the best possible start with using the system.

THE ROLE OF UAT

UAT is a test of an IS from the perspective of the users and other stakeholders for whom it has been built or acquired. UAT is not just a test of the software, nor of the functionality, performance, reliability and security, nor of the usability of the system. That does not mean UAT is not concerned with any or all of those areas, but that its primary concern and absolute focus is on whether or not the system can deliver the expected business benefits when operated by its designated users. All of the specialist areas mentioned will have, or certainly should have, been tested during development. None of those tests, however, answers the question ‘Is the system fit for purpose?’ UAT answers that question unequivocally by testing it against the reason it was built or acquired, using the eventual end-users as testers and, as far as possible, utilising the user documentation prepared to support their use of the system. This will entail not just exercising the system in some random or even structured way; it will entail using the system to enable business processes that deliver value using realistic (or actual) scenarios, data and operations. UAT is the nearest we get to ‘the real thing’ without actually taking the risk of releasing the system into service, and it is the exercise that will enable us to make a rational judgement about whether to take the risk of releasing it into service.

THE COSTS AND BENEFITS OF UAT

UAT is an expensive exercise. As well as the hard costs of actually doing the testing, we have to factor in the additional time that development resources are tied up before roll-out of the system allows them to be released to other projects.

The largest components of the direct cost of UAT are:

•  The cost of diverting end-user staff from their normal work to plan and carry out UAT. As well as salary costs there is the loss of whatever revenue the staff would have generated or the delay in realising that revenue.

•  The cost of training or familiarisation for the UAT team (and subsequently for all end-users).

•  The cost of test environments for UAT.

A UAT exercise with a team of six staff that takes two weeks to complete might have direct costs of around £6,000–£10,000 for salaries alone.

What do we have to offset this cost?

THE VALUE OF UAT – THE REASONS WE NEED TO DO IT

In case you are not convinced of the need to go to the time and expense of a UAT exercise, we will take a brief look at the world of ISs and what can go wrong with them.

Reason 1: risk management – avoiding expensive failures

Not all ISs end in high-profile failure; many are highly successful and trouble free over a long life, but it is nevertheless true that the statistical likelihood of failure for an IS is relatively high.

In 1995 The Standish Group published a report based on research into a large number of failures in software systems. It was called the CHAOS report (The Standish Group has never revealed what the acronym stands for, only conceding that a few select people in the organisation know) and it contained some frightening statistics, for example that only 16.2 per cent of IS projects were completed on time and on budget. Worse still, completion often required significant watering down of the original specification for these systems.

The statistics relate to system failures and these are not necessarily related to UAT, but the CHAOS report analysed their statistical evidence and identified key factors that contributed to failure. These factors point to problems in some aspects of system development that UAT can throw some light on, such as lack of user involvement and poorly defined requirements.

The original CHAOS report was published quite a long time ago, but The Standish Group has repeated its research and published updated results over a number of years. The results have gradually been improving with the introduction of agile project models and a greater awareness of the issues that cause projects to fail. The 2011 edition of the CHAOS report found that 37 per cent of all projects succeeded and Jim Johnson, Chairman of The Standish Group, stated, ‘This year’s results represent the highest success rate in the history of the CHAOS research. We clearly are entering a new understanding of why projects succeed or fail.’ Although the results show a significant improvement on the original 1995 figures, it is a sobering thought that, even with these unprecedented results, 63 per cent of projects in the study did not succeed. Although other researchers have criticised The Standish Group for providing little or no context to its figures, for instance by not distinguishing between projects where going over the deadline means failure and those where an reasonable overrun can be easily tolerated, nevertheless the CHAOS report presents a very useful list of the key factors that contribute to project failures, the knowledge and application of which can benefit any project. We shall explore some of these factors and how we can learn from them in Chapter 1.

Avoiding high-profile disasters or even low-profile mishaps is not solely about performing effective UAT of course. By the time we get to UAT the seeds of failure have not only been sown, they may well have already germinated; but well-planned UAT may still be able to highlight the potential for disaster before it actually happens. This is an important reason, but not the only one, for doing UAT.

Reason 2: confidence building – achieving expected business benefits

If we have acquired some expensive software or some software designed to do something really important for our business, we need to be sure before we accept it that we have minimised the risk of it not doing the job we acquired it for. This is not just a repetition of disaster avoidance, but a form of ‘good housekeeping’ that ensures we have addressed all the things that could go wrong when we start to use the system. Many of these may be only small things, but they can add up to a problem in service, and some of these smaller issues may not have been addressed by the testing done during development, which was designed to ensure the system meets its specification rather than ensuring that the users will be able to use it effectively. In Chapter 3 we will show how risk-based testing can be used to minimise the risk.

We will also need to be sure that the people who use the software will get the expected productivity or other benefits and that the business will get the efficiency or other improvements we intended. To do this we need to put the system to use in a realistic environment and work through some examples of situations in which it is expected to add value, such as streamlining processes or reducing the cost of production. This kind of testing will be based on an understanding of how the system will be used to gain business benefits and setting up a realistic ‘trial’ to see that it is capable of delivering those benefits. We explain in Chapter 6 how we can set up this kind of test.

Reason 3: assessment – getting the business processes right

As well as making sure that the system will not only do what it is supposed to do but will also be usable by our staff, we need to give the end-users an opportunity to exercise the system in whatever way they have agreed to work to make it effective. Users will need to check system behaviour as part of the overall business processes, using it in the way that has been agreed. We will not only treat the system as an IS and test its behaviour when the intended business transactions are passed through it, but we will also test that end-users can operate the system in the way that is needed to make those processes work effectively.

One further benefit of UAT is that the act of determining a system’s fitness for purpose necessarily involves comparing the system’s behaviour with user expectations. If the comparison is done systematically and formally, then the primary ‘result’ of UAT may be to accept or not to accept the system. A valuable secondary result, though, is that the comparison will have generated a detailed understanding of how well the system matches expectations. This comparison can be made the basis of an assessment of the system’s current capability and value to the business, which in turn can be used to underpin a strategy for future development or enhancement. It enables us to know exactly what the strengths and weaknesses of the system are and to assess value for money and potential for improving business value from the investment in development.

Reason 4: preparation – assessing readiness for service

Finally we want to be sure that the software is fit to release into the business and that the business is ready to make use of it, so we will need to assess readiness. After we have completed the testing based on risks, benefits and usability, we may have discovered some problems that need to be resolved, some defects that need to be corrected or some other aspect that needs to be addressed (such as training). The combination of these things will have an impact on when a system will be completely ready for the users to take on. This aspect of readiness is about how robust the system is in operation, how closely it fits expectations, how many ‘workarounds’ have been found necessary and a host of other factors that may affect whether or not the system is ready to deliver value to the business. This is usually a matter of discussion and negotiation, and results in a decision to release the software. Chapter 9 will explore the mechanisms we can put in place to make that decision as objective as possible. If we consider this aspect of UAT well in advance (as we should), we can define a set of acceptance criteria that will be used to come to a decision about release and so avoid the conflicts that can arise at this stage.

One important thing to say in this context is that UAT is distinct from training, familiarisation and all other aspects of roll-out. It has its own set of objectives that would be seriously jeopardised if UAT was carried out by unprepared, untrained end-users who were unfamiliar with the system. Having said that, however, the experience of UAT will be a learning exercise that is second to none and the experience gained during UAT can be captured and utilised in training, familiarising and preparing the end-user population for using the system effectively – including dealing with any issues raised by UAT for which ‘workarounds’ have been defined.

The enigmatic nature of software as a medium makes the risk associated with deploying ISs containing software significant and UAT is the best mechanism we have for managing that risk for the user community. We have already stated that the main reason why UAT is a worthwhile exercise is that it can help to avoid some of the costs of dealing with failures and of modifying the software after its acceptance. The potential costs of failure can be judged from the case study examples, but the cost of failure is invariably higher than expected in a business context because of the ramifications of failure in the tightly knit relationship between business processes and ISs.

There is also one other major benefit from UAT. The skills and newly acquired experience, combined with the invaluable experience that stakeholders bring with them to UAT, have enormous potential value in preparing for and achieving a painless roll-out. Beyond that, the insights gained by those spending time in testing a system can be harnessed to enhance future business process development, training and support. There is potentially a massive benefit to be reaped from the experience of UAT for the individuals concerned and for the business as a whole.

STAKEHOLDERS – WHO THIS BOOK IS FOR

This book is aimed at the groups of people who together comprise the principal stakeholders of UAT:

•  sponsors (typically owners or senior managers who have decided to acquire software of some kind to enhance their business);

•  business managers, who will be responsible for ensuring the IS delivers the expected business benefits;

•  end-users, who will be most interested in the way they are able to interact with the system to achieve their tasks;

•  professional developers and testers, who will be concerned with providing the most effective support for UAT;

•  managers of processes, practices and standards, who will take an interest in making UAT a cost-effective process using best practice.

In each of these groups there may be some who have no experience of testing at all and for whom an assignment to a UAT team is a daunting prospect.

If you are in this category then this book is for you, and it speaks to you very directly:

•  by using non-technical language wherever possible;

•  by assuming no prior knowledge of testing at all;

•  by identifying what you need to do in a step-by-step fashion;

•  by providing case studies, examples and exercises for you to practise at least some of the tasks.

If you are a sponsor and are in the process of acquiring a new IS, the book will identify the role of UAT in ensuring you get the system you are expecting, but it will also help you to understand how UAT fits into your overall planning and what benefits it brings in terms of training and supporting the inevitable change to working practices that new ISs invariably bring.

If you are a business manager who will be responsible for the effective operation of the system in achieving business benefits, the book will explain how those benefits are defined and measured and how you can ensure UAT is shaped to provide you with the information you need to prepare for the system’s release into service in your part of the business.

If you are an end-user tasked with testing an IS, the book will provide you with all the tools and techniques you will need to be an effective UA tester.

If you are in the group of ‘interested professionals’ who may need to support UAT, you will find in this book a clear exposition of how and where the world of UAT interfaces with the work of professional developers and testers and how best to manage that interface and support the UAT activity.

Finally, if your role involves managing and enhancing practice of some kind as a test manager, quality manager or project manager, you will find the book useful as a manual of key techniques, methods and tools. Quality managers wanting to improve UAT practices, for example, will find here a comprehensive review of UAT from which best practice can be distilled and then customised to fit your organisational culture, size and budget.

HOW TO GET THE BEST FROM THIS BOOK

The two main premises of this book are:

1.  We can and must make UAT effective to reduce the risk and cost of underperforming ISs.

2.  The benefits of UAT outweigh the costs.

To explain why and how these results can be achieved the book has been structured as described below.

The first three chapters provide all the background and justification for a step-by-step approach to UAT:

•  Chapter 1 – The Importance of UAT;

•  Chapter 2 – Business Requirements;

•  Chapter 3 – Testing Basics for UAT.

The next two chapters are dedicated to the practical issues of creating, shaping, training and deploying an effective UAT team:

•  Chapter 4 – The UAT Team;

•  Chapter 5 – UAT as Transition.

The remaining chapters explain the step-by-step approach in practical detail:

•  Chapter 6 – Preparing for UAT – Planning;

•  Chapter 7 – Test Design for UAT;

•  Chapter 8 – Implementing the Tests;

•  Chapter 9 – Evaluating the System;

•  Chapter 10 – Life after UAT.

We sincerely hope that all readers of the book will find it interesting and rewarding enough to want to read it from cover to cover. However, the first encounter with any book of this kind needs some kind of plan to extract the maximum value in the minimum time to provide a starting point from which a more detailed exploration of the content can begin. The initial route through the book naturally depends on your starting point.

All readers are strongly advised to read Chapters 1–3 as general background and as preparation for the practical approach later in the book.

Those responsible for managing a UAT team should read Chapters 4 and 5, and all those selected to take part in UAT will gain an understanding of the skills needed and the style of operation for a UAT team by reading these two chapters.

All readers are encouraged to read Chapters 6–10 although some chapters will be more relevant to some groups than to others. Within these chapters all of the stakeholders are considered and information of particular importance to one group or another is labelled appropriately to guide your initial reading.

CHECKLISTS

Whatever approach you take to reading the book you will be guided through all the key steps in UAT through Chapters 6 – 10. Your reading should help you to understand what the key steps are and why they are ordered as they are.

To help you in applying what you learn from the book, you will find a set of checklists for all the major steps in UAT at Appendix A. The purpose of the checklists is to present the key steps in a logical order, uncluttered by explanation, but also indicate where you can find supporting material in the book. We hope you will find these useful when you come to apply the step-by-step approach to your projects.

CASE STUDIES

Throughout the book we will make use of case studies, examples and exercises. The case studies are used to provide documented real-life examples of the things we are talking about; the examples will give you practical guidance on how to use the techniques, methods and approaches we present; the exercises will provide you with opportunities to try out some of the techniques.

In addition to these in-text components we end each chapter with a few multiple-choice questions that you can use as a check that you have understood the key ideas in the chapter and we also provide you with questions that you might like to think about. These raise some of the issues and challenges that you might face as a user acceptance (UA) tester and give you an opportunity to think about them before you have to deal with any real challenges in a project context. We have provided answers to all the exercises in Appendix B so you can check your answers against ours. The more open-ended questions at the end of the chapters have no correct answer, but we have included some of our own thoughts on these questions that you can read if you want to. We make no claims about these being the best responses to the questions, but they are the ones that we have developed from our experience.

Finally, before we progress on to Chapter 1 and take the first step in introducing the UAT process, we are going to introduce you to a small imaginary company called HMF that is acquiring an IS. We will use the HMF case study mainly to wrap around the more significant examples we provide, especially in Chapters 6–9, to provide continuity of ideas and hopefully to help you see how information is built up at each step of the approach for use in later stages.

We hope you find HMF and Excelsior, and all the case studies, examples, exercises and the questions at the chapter ends, helpful in bringing the ideas to life and enabling you to gain some insight into what is happening during a real UAT exercise.