APPENDIX B
ANSWERS AND COMMENTS
CHAPTER 1
Answers to what have you learned
Q1 B
Answer A is incorrect. UAT may take longer but usually takes less time than other testing.
Answer C is incorrect. UAT evaluates the system against business requirements (not necessarily captured in the requirements specification), business processes and user expectations.
Answer D is incorrect. UAT is formal testing.
Q2 B, D, E
Answer A is incorrect. Technical specifications are used for testing during development, but not for UAT because they have a technical perspective rather than a user perspective.
Answer C is incorrect. Business requirements are part of the basis for UAT but they may not be defined fully in a requirements specification.
Answer F is incorrect. There would be no value in repeating tests already run for system testing.
Answer G is incorrect. While tests could be automated they should not be designed by developers because UAT must be from an end-user perspective.
Q3 C, D, F
Answer A is incorrect. It may or may not be a true statement but it is not the reason users carry out UAT.
Answer B is incorrect. Users may or may not be experts on technical performance but it is not the reason they carry out UAT.
Answer E is incorrect. Developer testing is essential but it is different from UAT and both levels of testing are needed.
Our responses to questions to consider
1. What questions would you ask if your organisation asked you to carry out UAT on a new piece of software that is being introduced to support the sales activity in your business?
There are a number of key questions that the person conducting the UAT should ask, especially if there has been no previous involvement with the project. For example:
• What is the overall goal of the project?
• Who are the stakeholders?
• What stage of development is the project in?
• When will UAT (provisionally) take place?
• Is the project based on a waterfall or agile model?
• Is there a contract related to the acceptance of the criteria?
• Have the stakeholders been involved and have they bought into the project?
• Do the requirements reflect the current business need?
The aim must be to have a system that makes sales people more effective, so you will need sales people to be involved. They will probably be busy selling and will not want to run tests, but you will need to at least engage them in a conversation about what they need and expect.
2. How would you react if your boss told you that the development project for which you will be doing UAT is running late and he wants you to do UAT in parallel with the developers completing the development?
Running UAT in parallel with development is not only a waste of time and effort (because the system is likely to be changing from day to day), but also can provide a false sense of security. Even if the UA tests were passed successfully, that would be no guarantee that what was actually delivered at the end would work. On the other hand, there may be some value in having users familiarise themselves with the system during development – as long as this does not get in the way of development!
It is important for the success of the project as a whole that the value and purpose of UAT is understood as well as the entry and exit criteria. If the manager is not aware of the reasons for these restrictions, it is important to point them out. They may not realise that running in parallel with development will compromise the value of UAT and overall may be the source of greater delays.
3. Why not just get professional testers to do UAT? After all they have experience of formal testing and know all the techniques.
Successful UAT is the result of bringing together expertise about the business and its existing processes, expertise about the new system and how it will work, and UAT expertise. As the name suggests, the purpose of the testing is that the user accepts the system and this is not normally possible without the user taking part. Testing specialists carrying out testing will also introduce a bias in that they are likely to test how they know the system works as opposed to how the users will use it, simply because they lack the experience and the knowledge to recreate what a user would do. In some cases, where users cannot carry out UAT, it may be possible for users to specify exactly what ought to be tested to achieve acceptance and have others carry out the testing, although this will always represent a compromise to the quality of the UAT.
CHAPTER 2
Exercise 2.1
1. The requirement relates to all requests staff make for expenses, purchase orders, absence and so on.
2. The requirement does not relate to changes made by staff that do not require approval such as updating address or bank account details.
3. The requirement states that once the request has been made (saved) the requester should receive notification of any updates in status or other changes to the request.
4. The requirement is ranked in terms of importance but we cannot tell from a single requirement whether this is a ranking that relates to the whole RS, and is therefore the 34th most important requirement in the RS, or whether it is a ranking that relates to the section within the requirements it belongs to.
5. The requirement is of high priority.
Answers to what have you learned
Q1 A
Answer B is incorrect. Business intent is independent of UAT and would be needed even if UAT were not happening.
Answer C is incorrect, although it is another important aspect of the system that is tested at UAT.
Answer D is incorrect. The purpose of the business is the overall reason for the business to exist. Business intent is the reason for building or changing an IS within that business.
Q2 C
Answer A is incorrect. Business requirements are a significant part of the test basis for UAT but the strategy defines how we will test for acceptance.
Answer B is incorrect. Although business requirements form part of the UA test basis they are not the complete test basis (user expectations form another part of the UA test basis).
Answer D is incorrect because answer B is incorrect.
Q3 D
Answers A, B and C are all possible limitations of requirements, so answer D is the best answer.
Q4 B
Answer A is incorrect. Requirements need to have enough relevant detail to make them clear, although unnecessary detail must be avoided because it will make the requirement unclear.
Answer C is incorrect. The level of technical detail will vary between requirements but requirements are about user needs so should be expressed in user terms. This does not necessarily make them non-technical (for example the user may be an engineer and need technical details in the requirements) but technical detail should be kept to the minimum consistent with clarity and readability.
Answer D is incorrect. It is a description of a good requirement.
Our responses to questions to consider
1. You are a stakeholder new to the project and are asked to read the RS to get an idea of what the project is about, but you do not understand some, most or all of the requirements. What would you do?
It would be a good idea to check first with your teammates and team leader to see what level of understanding you are expected to need. If there has been specific training set up for UAT, you would clearly benefit from it; if not, you might like to suggest that something is put in place. If all else fails you can consult the users, developers and business analysts as appropriate, but remember they are likely to be busy people.
You will need a good understanding of the requirements to be able to design and/or implement tests. If you think this might be an issue you need to flag it.
2. What is the difference between an RS and a UA test basis?
An RS is the formal expression of requirements that has been documented for development and that may have been subsequently updated. A test basis for UAT is the set of documents needed to enable UA tests to be designed and implemented. The test basis will need to include a definition of user expectations and a description of business processes as well as an up-to-date version of the business requirements with any changes made since the document was initially approved. So a UA test basis typically updates and extends the RS.
CHAPTER 3
Answers to what have you learned
Q1 A (strictly A is functional coverage but it is the only answer that relates to the test coverage idea)
Answer B is incorrect. It simply counts total tests run and has no direct connection with coverage.
Answer C is incorrect. Tests are normally designed to achieve a single test objective.
Answer D is incorrect. Scope is the breadth of testing, which is a different kind of measure of what is and is not tested. Test coverage measures how much of the in-scope functionality has been covered by testing.
Q2 D
Answer A may be true but is not the correct answer because it is not a benefit of reviews as a means of evaluating documents.
Answer B is incorrect. Reviews are not inexpensive; in fact they soak up valuable effort, which is why they must be effective in finding defects.
Answer C is incorrect. Reviews can only expect to find some of the defects but enough to make the effort worthwhile because the cost of finding the defects any other way or of not finding the defects would be higher.
Q3 B
Answer A is incorrect. Test techniques exploit what happens when users enter non-valid data to identify defects but do not affect what users can enter.
Answer C is incorrect. Boundaries are the edges of partitions rather than extreme conditions. Some boundaries may be at extremes, for instance the highest value a system can handle, but the test is a boundary test because it is at the edge of a partition rather than because it is using extreme values.
Answer D is incorrect. Only those test cases that are outside a boundary should fail, while those inside or on the boundary should pass.
Our responses to questions to consider
1. Your organisation is reluctant to allow UA testers to be part of a review process. What would be the best way to overcome that reluctance?
It would be useful to find out the cause of this reluctance. Is the worry caused by a fear of overcrowding or a fear of certain objections being raised that may delay the project? The original CHAOS report showed that the second most common cause for project failure was lack of involvement. The review process aims to make sure that the requirements match the needs of the business and the business ought to be involved in this process.
Even if the other stakeholders are part of the review, users or a user representative in the form of a subject-matter expert being present will help to achieve the aim of the review. It may be useful for everyone involved in the review to understand its aims and to have the message reinforced that no decisions need to be made in the review as to what changes are required to the system, only to establish what the current need is.
2. You are being offered a training course before starting work as a UA tester. What would be your requirements for a one-day course? Suppose you could have three days of training. What changes would you make to your requirements?
This is a tough call but worth thinking about. We’ll be offering some help in Chapters 4 and 5 but now would be a good time to think about what you would need to cope with UAT in your own organisation. If you have a look at the content of publicly available courses, for example, you can get an idea of what is typically delivered and identify what would be gaps for you. Areas such as getting to know the system you will be testing, understanding how your development team has tested the system and becoming familiar with any tools you may be expected to use are all areas that generic courses cannot provide. You will need to find a way to get this information for yourself.
Timing is everything. One day is not very much time to fit in all that you might want to learn about to feel ready to test, but it is still a good idea to distil your ideas down and see what could be fitted into one day. Then give yourself the luxury of three days and consider how you might do it differently. These thought experiments will set you up well for the material in Chapters 4 and 5 on building and training an effective UAT team.
You already have the most important knowledge because you are a user. Probably the best value for your one day would come from gaining familiarity with requirements (take some of your own to make sure you get some practical value from the training). The second thing would be to learn how to get the best from reviews. You can pick up some basic testing knowledge from books.
In a three-day course you could add some testing basics, but the main benefit of the extra time would be to enable you to get some practice in reviewing and in writing test scripts (if possible, based on your own requirements).
CHAPTER 4
Answers to what have you learned
Q1 A
Answers B, C and D are all incorrect because these are roles that are external to UAT, although they need to have a very close relationship. They all have wider responsibilities but we would expect them to provide help and support as needed.
Q2 A
The correct sequence is forming, storming, norming, performing.
Q3 B, D
Answer A is incorrect although still very important. Written job descriptions should provide everyone with a clear sense of what is expected from them, which is important to team effectiveness.
Answer C is incorrect. It may be appropriate for those with technical skills to take the lead on occasions but that is usually in the interests of progressing the task rather than building team spirit.
Answer E is incorrect. It is an important rule for leaders to be clear and decisive, and a good leader has the best chance of building an effective team, but leadership skills are part of separating the leader from the team rather than building team spirit.
Our responses to questions to consider
1. Your most knowledgeable UA tester has been identified as a complainer. What would you do?
There is a choice here between trying to ‘keep the peace’ and taking an opportunity to build the team’s morale. If you ignore the complaints, you could demotivate a good tester. If you respond to the complaints (perhaps at the expense of time to do other important things), you give the complainer a sense of increased importance and perhaps create tensions within the team. Keeping the peace options are fraught with danger.
If you spend some time with the complainer, one to one, and try to understand the underlying problems that lead to complaints, you may discover genuine issues that need to be resolved. One way to move the situation forward might be to give the complainer the opportunity to be the team’s spokesperson in resolving the issues on behalf of the team – and of course tell the team to take their issues to the new spokesperson.
2. One team member is being regularly distracted by their line manager to do routine tasks. How would you deal with this situation?
This is a common challenge. It points to one of two possible scenarios: either the team member is genuinely too busy in their ‘day job’ to be able to commit enough time to UAT or the line manager is unwilling to manage without a staff member and is asserting their authority to make life difficult for the team. Either way a frank discussion with the line manager should uncover the truth and provide a way forward. If the problem really is overload, a replacement should be sought and the original team member released. If the problem is one of attitude, it needs to be challenged and, if necessary, escalated.
3. Your company does not have space to house the UAT team away from the rest of the business but is willing to give the team half of each day to work on UAT. How would you organise the team to work?
This is far from ideal but it does at least provide an opportunity for some daily team time. If a suitable location can be found, there is an opportunity to create some ‘team space’. Somewhere like a local community hall or even a back room in a pub (that is not open all day) can be a good meeting place. There will be a small hire cost but for that you may have the chance to store things at the venue so you can make the place feel like the team’s home.
4. Your company provides a room for the UAT team to work in but takes the view that all your normal work must be completed as usual during UAT. How would you ensure that the UAT meets its deadlines?
Clearly the team needs a routine so that everyone is available at the same time when necessary, although there may equally be times when we want to stagger the working hours. During planning, for example, we will want the whole team together. During test execution it may be more attractive to have an extended working day with the team split in half, with a handover in the middle of the extended day. This will require good team discipline so that time is not wasted, and good cooperation with the business to ensure normal work is not being neglected, so regular communication between the team leader and the rest of the business is important.
CHAPTER 5
Our responses to questions to consider
The project you are working on is acquiring a COTS-based suite of applications configured to meet your company’s needs, but there are some known potential gaps in functionality. Implementation is scheduled for 12 months ahead and a UAT team will be formed 1 month before UAT.
a. As UAT team leader elect you have been asked to review the plans. What feedback would you give?
There are a few challenges here. The COTS approach means a contractual acceptance as well as an internal one, and payment will have to be made on contractual acceptance so it needs to be thorough. The team will need to get up to speed with the COTS modules and also have a strong grasp of the requirements before UAT, which points to early training and a start on the requirements review and learning at least three months before UAT. It is all a question of risk. Is it worth forming a team earlier to get a good start or take the risk of accepting a system by default because testing is not good enough to demonstrate weaknesses?
b. As a nominated end-user tester for the UAT team, what would you propose?
As an end-user I know I’m going to be asked to continue with my ‘day job’ while I prepare for UAT. In that case the earlier I can start on the preparation work the better so I can stagger it over a longer period. I will need some training just before I start the requirements review so I would propose that training is arranged for the team as soon as requirements are available for review, then the team is given an allowance of time over a period to get the review done and begin preparing the tests.
CHAPTER 6
Exercise 6.1
A4.0 – Ambiguous, ‘colleagues’ should be replaced with some reference to the user hierarchy, for instance members of the team as defined in the user hierarchy. Contains two requirements: that the calendar is visible to the user, containing the right information, and that it must be visible at a particular point in the absence process. Too focused on how the solution ought to work.
A4.1 – Can be broken down into two requirements: that the system should show the existing balance of remaining days off and that the system should recalculate the new balance after placing the planned days off in the calendar.
A4.2 – OK.
A4.3 – OK.
A4.4 – Ambiguous, should refer to what the holiday year is, for example January to December or a year from the start date of the user.
A4.5 – Poorly defined, reference to other requirement without referring to the unique ID.
A4.6 – Contains two requirements.
A4.7 – Poorly defined, should refer to the user hierarchy and contains multiple requirements.
A4.8 – Ambiguous, should not contain the word ‘soon’, but rather define a specific time period. Focused on the solution as opposed to the need in referring to a ‘system reminder’.
Exercise 6.2
There are at least three other potential user stories that could have been used in the case study example but were not for the sake of brevity. User story 1.1 is fairly straightforward and represents existing functionality within the system. UAT will now include the tests that ensure someone logged on with a manager role will be able to send expenses for approval. There may be other managerial roles that need to be tested that were not included. User stories 2.1, 2.2 and 2.3 relate to assigning an account so that one user can manage tasks on another user’s behalf. User story 2.3 hinted at the fact that the manager and assistant relationship may not be the only one that could benefit from the ability to assign accounts. Other user stories could be created around assigning accounts to deal with absence for a number of different roles in the organisation. User stories 3.1 and 3.2 deal with third-party billing and how this is recorded on the accounts system. In the case of third-party billing, additional contract details may be required, for instance the third-party address. If this is the case, the system will have to deal with what happens to the third-party billing address if the contract is subsequently changed as is described in use case 3.2. There may also be other roles that need to be able to select or deselect the third-party billing option and may have to be able to edit the contract in other ways, specifically related to the tax for the purposes of the accounts payable department.
Answers to what have you learned
Q1 C
Answer A is incorrect. This is an impossible objective and needs to be qualified (for example no critical defects).
Answer B is incorrect. This is also an impossible objective to achieve because not all risks will be known and it will not be possible to eliminate all known risks. A more reasonable target would be to reduce the known risks to an acceptable level.
Answer D is incorrect. It may not be possible to complete everything in the UAT plan as originally agreed. Some aspects will be essential (for example achieving acceptable test coverage) and these need to be specifically stated.
Q2 D, E
Answer A is incorrect. It may not be possible to complete all planned testing but the testing completed may provide adequate test coverage.
Answer B is incorrect. Change requests can come in at any time and will continue to arrive after release of the system. They have no bearing on release unless a change is deemed essential, and that would only arise in UAT if it was related to a critical defect of some kind.
Answer C is incorrect but tempting. Normally all test incidents would be at least investigated before release, even if not cleared. Technically a test incident is outstanding until it is cleared, but if investigation showed, for example, that the failed test was due to an error in the test script there would be no reason to delay release.
Q3 A, D, E
Answers B and C are incorrect because they are techniques for expressing requirements and not requirements in themselves. Typically user stories might be used to capture undocumented user expectations and use cases might be used to capture undocumented business processes.
Answer F is incorrect. Technical specifications would be part of the test basis for system testing but not for UAT.
Our responses to questions to consider
1. You are finding that users are reluctant to express ideas about what they expect from the system when it is delivered and fall back on what is in the requirements. What would you do?
The reaction is common where users are either not fully engaged with the changes that the new system will bring or are not well informed about what will happen. They may be expressing lack of interest or lack of knowledge. Either way it would be useful to try to find out why because the outcome could be a system that they are unhappy with or cannot use.
2. The sales manager believes that acceptance criteria should not delay delivery to customers who have expectations of early delivery. The development manager believes that acceptance criteria should include only requirements coverage. The marketing manager is concerned about the image of the business and insists on zero serious defects as an acceptance criterion. What should you do?
Conflicting priorities are not unusual but they need to be harmonised or you will have no secure basis for acceptance. A workshop involving all the stakeholders can be a very good vehicle for flushing out such disagreements and getting a resolution.
All three of the roles are seeking to minimise the impact on their own area of responsibility and do what they believe is best for the business. There is always a compromise position, especially if the individuals are willing to work together to resolve any problems and minimise impact for each other.
3. The development team is reluctant to give the UAT team access to the incident reporting system because it could result in lost incident data if someone makes a mistake. What would you do?
An understandable concern. Until the system is released into UAT the development team has ownership and control, but this might be a good time to ask the development team to provide some training on the tools so that end-users are less likely to make mistakes when they use them. The training may build confidence enough to encourage the development team to allow access before UAT begins, or you may discover that additional controls are needed to prevent any data loss.
CHAPTER 7
Question 7.1
There is no link between the two test scripts that makes it logical to carry out the first test and then the second using the same UA tester. The output from one script is not equal to or part of the input for the second script. In essence both are checking the same screen for the correct information but logged in as different roles. If there is time to do so, UAT should test the whole contract process end to end. It may therefore be useful to have one tester logged on as the team member and one as the team manager (ideally testers who are a team member and team manager) where they use the same contract, created by the team member, and follow it through all the stages of the process together. The team member can send the contract for approval and the manager can approve it, while they both check that the screens used are correct according to the requirements. A number of alternative processes may be tested including rejection of the contract, which represents an alternative path. If the test scripts were carried out in sequence by the same tester, the tester would have to log off as the team member and log on again as the team manager, which would be time-consuming and also does not, in our example, represent how users will use the system in real life.
Question 7.2
There are an almost infinite number of entries that do not meet the criteria of the example of the allowed entry and therefore represents a negative test. The number can be limited to certain types of negative tests and refined by using user- or tester-generated scenarios that represent the kinds of mistakes users are most likely to make:
1. Wrong number of characters; according to BVA we may try 0, 1, 6 and 7 characters.
2. Leaving a required field empty; already covered by the zero characters entry in the previous example.
3. The wrong type of character, * & = -; we want to try to find the most likely candidates, for instance a foreign alphabetical character such as é.
4. A capital letter.
5. Any character produced by inserting a valid character but with the shift or control key depressed at the same time.
Answers to what have you learned
Q1 A
Answer B is incorrect. Traceability is a mechanism and not a measure.
Answer C is incorrect. Traceability could ensure that there is a test of each requirement but not that the test data are complete.
Answer D is incorrect. It is important for traceability that each requirement has a unique ID but traceability is not the means of achieving it.
Q2 C
Answer A is incorrect. This is a test precondition.
Answer B is incorrect. Collections of features are not called test conditions.
Answer D is incorrect. Exit criteria are not the same as test conditions.
Q3 A, C, D
Answer B is incorrect. Tester availability is not related to test design but is a planning factor.
Answer E is incorrect. Risk is a key driver of test strategy and may be used to prioritise tests but individual UA test cases are mostly directed at requirements, processes and user interfaces.
Q4 C
Answer A is incorrect. BVA does help to minimise the total test cases but not all of these are negative.
Answer B is incorrect. BVA does not identify all negative test cases. In fact no technique can do that; the number is potentially very large.
Answer D is incorrect. Boundaries identify the edges of partitions but BVA is applied to test at the boundaries, not to identify them.
Our response to questions to consider
1. The test cases have been created. When deciding what order they should be placed in, the project manager wants you to focus on risk and the project sponsor wants you to focus on processes. How do you decide what order they should be placed in?
It is most important to know what the project manager means by risk and therefore what risks they want to mitigate. There may be a part of the system that is key to success or failure, will have the biggest financial impact if it fails or that may be a part of the system that is seen as high profile. The business processes may already cover the high-risk and high-profile functionality. If they do not, ensure that the project manager and the other stakeholders come to a consensus on the issue of what ought to be tested first. If the parts of the system that are considered the highest risk are part of the day-to-day processes, the two approaches can be combined, along with any other considerations you would like to include, to create an approach that meets the criteria of the project manager and the sponsor.
2. On the first day of UAT half the team are off ill with a cold. What decision would you make about whether to run testing? What tests do you think you could run?
You may need to do a little investigation and separate the processes that require input from the whole group or that are complex and need a number of different users to test together from those that are simple or that stand alone. If this work will take a lot of time or would be very disruptive to testing, UAT may need to be postponed.
CHAPTER 8
Exercise 8.1
1. In our example, in test script #1 the tester is logged on with a user role that is not allowed to access the Contracts module and in test script #2 the tester is logged on with a user role that is allowed access to the Contracts module. It would not be efficient to have these test scripts follow one another in the same scenario because it would involve logging off and logging back on again. It would be more useful to add each script to a set of tests that require this log on.
2. If possible, the tests should be included in the process or processes that cover the contracts functionality. As different roles are required to test this process and those roles have a part to play in progressing it, the check for the logon to the module could sit at the start of the contracts process tests for those different roles. Because they need to log on to access the module in order to use it, they can test this functionality at the same time.
Answers to what have you learned
Q1 D
Answers A, B and C are not incorrect but only answer D is complete.
Q2 D
Answer A is correct but incomplete.
Answer B is incorrect. Since the test script has been created it is part of the formal test documentation and we will need a record of the test failure and subsequent correction.
Answer C is incorrect for the same reasons as B.
Q3 B
Answer A is incorrect. This is an estimate.
Answer C is incorrect. Incidents can be recorded on a test log but they are not the only information recorded.
Answer D is incorrect. As with C, this is correct but not complete.
Our response to question to consider
1. Halfway through testing you are asked to provide a fixed date for finishing UAT. What would your answer to this request be?
It is possible to provide an estimate of the time required to finish UAT based on the time it took to complete the first half of UAT. It does not provide a guarantee, because issues may arise that are unexpected and perhaps critical that did not occur in the first half of the testing. If a fixed date is required, it comes with the risk that not all of the testing will be completed. It is important to establish that this risk exists and that it represents a smaller or otherwise more acceptable risk than missing the deadline would have. If the acceptance criteria include a coverage percentage that is required for UAT sign-off and the omitted testing affects the coverage percentage that can be achieved, the acceptance criteria also need to be adjusted.
We should have learned from the first half of UAT. If any delays were due to incidents and defect corrections, we could estimate the likely impact of a similar level of incidents in the second half of UAT (or scale the impact according to whether we think the second half is likely to have more or less challenges). If delays were caused by shortages of staff, equipment or skills, we can identify whether these problems have been resolved and estimate accordingly.
CHAPTER 9
Answers to what have you learned
Q1 D, E, F
Answer A is incorrect. This is part of the decision logic but not one that determines acceptance.
Answer B is incorrect. This may well be a challenge but it does not automatically determine acceptance.
Answer C is incorrect. This is also part of the decision logic but does not determine acceptance.
Q2 D
Answer A is incorrect. A risk-based testing strategy helps to mitigate risk but if testing is finished early, the strategy may not have been completely implemented.
Answer B is incorrect. Continuous evaluation is important to identify progress towards acceptance criteria but this does not mitigate risk if testing is finished early.
Answer C is incorrect. All activities in UAT contribute to the eventual risk outcome. Identifying a risk-based test strategy, for example, is a key step.
Answer D is the best answer because it combines a risk-based approach with continuous evaluation so that risk mitigation can be monitored and an informed judgement made if testing has to stop.
Q3 D
Answers A, B and C are all more or less likely. If UAT has progressed to a risk assessment then it is very unlikely to be seriously flawed; the cost of continuing development and testing to this stage would be very high for a system with major problems and action would have been taken earlier, so outright rejection is unlikely.
Our response to questions to consider
1. The test manager wants to set up a meeting to discuss the release towards the end of UAT. Who should they invite and why?
There are two sets of interested parties, those involved in UAT and the stakeholders. Representatives from both groups should be present. The meeting should ideally not be the first communication on the success and the progress of testing. Those involved in UAT should be prepared to interpret the UAT results in a way that is meaningful to making a release decision, and the stakeholders should be prepared to make a decision based on the data. The more relevant points of view that are present, the more likely the meeting is to reach a suitable conclusion.
2. There are a number of critical defects still outstanding. What does this mean in terms of the risk of release and to the release decision?
It would be hard to come to any conclusions about the release of a system while critical faults are unresolved. The only release decision that should be made is to either not release or to defer releasing the system while the critical errors are fixed. It is also hard to assess the progress of UAT in these circumstances as critical errors usually stop other testing from taking place and regression testing is also likely to be required. The risk of release at this stage is high and the system will not work as it was intended or in any other meaningful way.
The questions and comments represent a useful resource for the creation of FAQ content and will provide valuable information to support business as usual (BAU) staff as well as the end-user trainer.