4Managing Across the Organization

Keywords: None

4.1Introduction

No matter what type of organization you work in, or what your management reporting structure is, there will be managers across the organization that you must work with. No matter how effectively you do all the other tasks of test management, you will not be seen as successfully managing testing unless you work effectively with your fellow managers outside of the test team. This is often referred to as outward management or as Peter Drucker put it, managing your managers. Even if you are quantitatively effective and efficient in terms of the metrics discussed in Chapter 1, other managers’ perceptions of your capabilities are also an important measure of your capabilities.

These managers include your peers and colleagues in development, release engineering, technical support, and business analysis. They also include your direct supervisors, senior business and technical management, and executive management. Testing involves inputs from other groups, and these needs must be communicated effectively to the managers in order for you to obtain the inputs in a way that supports testing. Testing generates information that must be shared with other managers in order to deliver the value of testing to the organization.

In this chapter, we’ll examine important considerations related to managing across the organization.

4.2Types of External Relationships

In every case in which we have assessed test groups and their value to the organizations they serve, and in our own personal experiences as test managers, we have found that testing saves far more money and effort than it costs. Much of this savings is due to the escalating costs associated with customers and users finding defects compared to the relatively cheap cost of finding and removing defects in testing. Sadly, it is often true that this value is not fully understood by the others in the organization. Therefore, the test manager’s job includes promoting the test team and its value across the organization.

In addition, in most of our assessments, and in our personal experiences as test managers, we have found that test groups can increase the value they deliver to their organizations. This often involves finding ways to get more closely integrated and involved with the entire development process, especially in early testing activities such as requirements reviews. However, this is often perceived as an extra cost and sometimes even as a pesky interference. Therefore, the test manager’s job includes advocating the test team’s role—and the expansion of it.

Finally, in most of our assessments, and our personal experiences as test managers, we have found that testing is often a controversial activity. In some cases, this is due to an unfortunate and erroneous perception that testing is an adversarial activity, one that targets the developers to make them look bad. Even when this dynamic does not exist in an organization, it can be controversial to say that some feature or characteristics of the system under test does not work properly. People might become upset at the testers involved in the testing when such defects are reported. In these situations, the test manager must defend the test group. The test manager must always look for ways to position testing as constructive and valuable.

4.2.1Promoting and Advocating the Test Organization

Part of any manager’s job is marketing their team within the organization, because no manager—at least none that we’ve ever met—had the power to approve their own budget. Managers must sell the value of their group’s work to the people who fund it and to the people who receive that value. With test management it’s no different, but the job is a little tougher because testers don’t produce saleable products. What we do produce is information: assessments of the quality of the test items, measurements of our progress in testing, surrogate measures of how confident the organization should be in the software. We’ll discuss this information, and how to deliver it, further in Chapter 6.

Information can be miscommunicated. Information can be misunderstood. Information can be taken for granted. Information can be frustrating when it confounds plans. Information is intangible. Our products and their values are more subtle than those of many other groups in the organization, who deal in saleable software products. You must identify the value of testing and the people who receive that value. We’ll talk more about how to do that in the next section.

However, before we get into the specifics, let’s talk in general about promoting and advocating a test group, or really any group. Here are four important elements in marketing and selling a test service, whether to your colleagues down the hall or to clients around the world.

Demonstrating the benefits: Be sure you capture the quantitative and qualitative benefits of testing, as we will discuss in the next section. Identify the recipients of these benefits and discuss the benefits with them. If you have saved the company 1,000 person-hours in terms of avoided costs of field failures, the technical support manager, managers of teams that use or sell the software, and your own managers should know that. Notice that we are talking about external recipients of these benefits. If your group does something that benefits your group internally but has only indirect benefits to external groups—e.g., having monthly team-building lunches to build esprit de corps—these activities are not your best candidates for promoting your group. Also, be careful to be honest, unpretentious, and realistic about the benefits you promote. Making bloated and ridiculous claims about the benefits testing is delivering will undermine your credibility and the cause of the test group.

Emphasizing the successes: In addition to the ongoing benefits, when your test group accomplishes something worthwhile or achieves a major milestone, make sure to let the appropriate people know. If your automation team finishes a complete set of regression tests in place for a major software system produced by your organization, tell the managers in charge of that system and those who either use it or sell it to customers. Be sure to explain this accomplishment in terms of the benefits it offers to the organization. You can tell people, “We have 7,547 automated regression tests for the BusyWhack system,” and that’s likely to be received with a polite smile or maybe a more pointed, “So what?” Instead, tell people, “We have on average 25 automated regression tests for every functional requirement for the BusyWhack system and can now, in a weekend of automated testing, reduce regression risk to the same low level that used to require four weeks of five testers doing manual testing.” And again, remember to be realistic about the benefits you claim for these successes.

Publicizing upcoming improvements: When your team is working on a major improvement, inform the people who will benefit from it. If you are introducing a new test management system that will provide better and faster test results reporting and make the results accessible to all project stakeholders, talk to people who work on projects and can benefit from the improved reporting. Make it a conversation too—not just an announcement. What do people want to see in the way of test results reporting? How can your new system better serve people’s information needs? If you haven’t tried it, you might be surprised how helpful it is to ask stakeholders, “Here’s what we’re currently doing to improve this part of our test process. What improvements would you like to see?”

When necessary, admitting to and fixing mistakes: No one is perfect. Mistakes happen. Remember, the whole reason your group is there is because of the mistakes made by business analysts, system designers, database administrators, programmers, and other people engaged in creating the systems you test! So, when people in your group make mistakes, find the root cause of the mistake and fix it, if you can. If you need help from people outside your group to fix the root cause, approach them forthrightly and ask for their help. If people outside your group notice the mistake—or are impacted by the mistake—take responsibility and explain how you have fixed it—or have a plan to fix it. Trying to hide the mistake, or laying the blame elsewhere, will often do more harm than good to your reputation and the reputation of the test group.

Now, marketing and selling a service are good and important, but you also must be able to perform the service. This means that part of advocating the test group includes advocating to your direct managers and upper management the need to maintain a capable test group, one that can deliver the benefits, achieve the successes, continuously improve, and fix mistakes when they happen. Here are five important elements in maintaining a capable test group:

Adequately staffed: We discussed test objectives and test goals, and test strategies to achieve those goals, in Chapter 1. Having a strategy for success is important, but you will need enough people to carry out the test strategy. We’ll discuss how to estimate the number of people needed in Chapter 5. Understaffed test groups simply bounce around from one project to the next, and often one crisis to the next, perhaps sufficient to do some testing and make some incremental difference, but such a group cannot be—or be seen as—an important, strategic part of the organization’s quality strategy. Relevant to the topic of this chapter, you will not be able to credibly promote such a group as central to achieving quality; you will need to start by advocating sufficient staffing to your managers.

Well-trained: Not only do you need enough people, you need competent people. As was discussed in Chapter 2, a strong contributor can be 10 times more capable than a weak contributor. You might say, “I don’t have a team of strong contributors, but they’re about average for their roles, so what’s the problem?” Well, a strong contributor is about two and a half times more capable than an average contributor, which means you can effectively double the capability of your test group by focusing on increasing skills.¹ Part of that skills increase involves training, and getting training funding is a key part of making that happen. So you need to advocate adequate funding to build a strong test group.

Properly resourced: Even an adequate, trained test group can do little without the proper resources. Test environments, test tools, test management systems, defect tracking tools, workstations, reliable and fast Internet access, and good communication facilities are among some of the resources needed to do the job of testing. The resources required for testing are often expensive, so once again effective advocacy and promotion of the test group, and its successes and benefits, by the test manager is often necessary to secure these resources.

Well-informed: As we’ve mentioned before, testing produces information. For that information to be timely, credible, and useful, the people producing it must have the information they need to do their job. Testers need adequate test bases and test oracles to create and execute tests. Testers need to understand the intended production or customer environments to set up test environments. Testers need to be up-to-date on current project developments to respond currently to changing circumstances. If the requirements change during testing, testers must know that.

Respected: This final element is deeply intertwined with the other four. When a test group is respected, all four of the other elements will either follow directly or at least be more easily obtained by the test manager. When a test group is not respected, all will be more difficult to obtain. Respect is not given; it is earned. Part of earning respect is dealing respectfully with others; the test manager and all members of the test group must do so. In addition, the test manager must look for opportunities to advance the test team and its capabilities through the elements discussed in this list and the one preceding it. The test manager should continuously evaluate ways to improve the test group (e.g., through skills growth as discussed in Chapter 2) and look for ways in which the test group can contribute more to the organization.

These elements of marketing and selling a capable test team need not be a matter of persuasion and politicking. Fortunately, it is possible to objectively quantify the value of testing and to put forward solid, direct, qualitative benefits too. These values can be used both statically, to define the value currently delivered, and dynamically, to forecast the value that can be delivered. These forecasts can be based on proposed process improvements and possible paths forward for current projects. This predictive power is yet another value of testing.

Let’s see how the test manager can accomplish these goals in the next section.

4.2.2Selling the Value of Testing

As you just saw, part of the role of the test manager is effectively communicating and selling the value of testing. As the president of a global test consulting company, this marketing and selling role is something Rex has become quite familiar with, but he’s also learned that test managers within an organization must be no less adept at this skill.

As discussed in the Advanced and Expert syllabi, values of testing are both quantitative and qualitative. They can be delivered to the organization as a whole, to projects, or to ongoing operations. Let’s look at some of these values.

Finding defects: Since the cost of repairing a defect tends to increase the longer it persists through the software lifecycle, this value of testing can be quantified. Later in this section, we’ll look at an established technique, cost of quality, that can allow you to do so. This can apply both to defects found during dynamic testing, such as system test or system integration test, and to defects found during static testing, such as requirements and design reviews. For defects found and removed by static testing, the effect is magnified because these defects are prevented from amplifying their cost and number by escaping into the coding process.

Reducing risk by running tests: This value can be quantified through a calculation of the likelihood of certain types of failures along with the financial impact those failures would have if they occurred in production. Because of the difficulty of obtaining reliable numbers for likelihood and impact, this value is typically discussed in more abstract and qualitative terms. It is important to remember that the level of quality risk cannot be reduced to zero through testing, due to two principles discussed at the Foundation level: the impossibility of exhaustive testing and the ability of testing to demonstrate the presence of defects but not to prove the absence of defects.

Delivering information: The test process delivers—or at least should deliver—information about project, process, and product status that leads to better decisions. Given the high percentage of projects that fail, and the fact that these failures often result from bad decisions, this value can be quantified by looking at the percentage of similar projects that fail and the potential loss if this particular project failed. As with risk reduction, it is usually difficult to obtain precise numbers for likelihood and impact, so this value is typically quantified with a rough estimate or simply discussed in abstract and qualitative terms.

Building and increasing confidence: The more thoroughly tested a product is, the more confidence we can have that fewer undiscovered bugs remain. This value can be somewhat quantified through various coverage measures, such as risk coverage, requirements coverage, configuration coverage, interface coverage, data coverage, code coverage, and other relevant coverage measures. If such quantification is attempted, three caveats are important. First, different relevant measures apply to different products and test levels, so carefully select the coverage dimensions used as a surrogate metric for confidence. Second, similar with risk reduction and for the same reasons, do not let people believe that 100 percent coverage means 100 percent confidence that no undiscovered bugs remain. Third, due to another testing principle, the absence of errors fallacy, a high level of confidence in the quality of the product does not mean people should expect automatic success of the product in the market or with users.

Improved reputation for quality: Organizations that have experienced quality problems on past projects, especially when quality problems affect customers and users, often institute or improve testing groups in a desire to improve their quality and thus their reputation. Indeed, this is practically the mother of all testing values, in the sense that very few organizations would bother with testing if they didn’t feel that testing had some ability to improve quality. This value also has the potential to be enormous and wide-ranging. Higher-quality products enjoy improved customer satisfaction, and those customers have increased goodwill toward the producer. This leads not only to avoidance of lost sales due to quality problems, but also to increased sales due to positive word of mouth. A reputation for poor quality has the opposite effect, of course, and has been known to create serious, even existential problems for products. Be careful not to oversell this value, though, because so many other parts of the software engineering process affect the quality of the product. The best testing group in the world cannot save a fatally flawed product. This value is best sold as a qualitative value because it is unlikely that there is any meaningful way to quantify it.

Smoother and more-predictable releases: Similar to the value of improved quality, organizations that have seen quality problems affect their release plans and schedules, especially defects discovered late in a project, may institute or improve testing groups. Indeed, testing can help in this regard. Risk-based testing can result in important tests being run earlier in test cycles. Early testing activities (another principle mentioned in the Foundation syllabus) can reduce the number of defects that need to be found in later testing activities. However, as with improving quality, there are limits to what you, as a test manager, can do to deliver this value if other parts of the software engineering process are not arranged in a supportive way. This value also is best sold as a qualitative value because, again, it is unlikely that there is any meaningful way to quantify it.

Protection from legal liability (compliance with regulations, standards, and contracts): This value is a variant of one or both of the previous two values that applies when an organization has a legal responsibility for the quality of its products. Two common examples are regulated products, such as medical devices, and custom development for a client, at least a client that has been smart enough to include appropriate wording in its outsourcing contracts. This is an interesting value in that 100 percent coverage of specified requirements, along with complete adherence to other applicable contractual agreements or standards, can be sufficient to provide almost bulletproof protection. (We say “almost” because situations that result in death or serious injury will usually result in legal complications. As the lawyers’ aphorism puts it so well, you might beat the rap, but you won’t beat the ride, and that ride can be very expensive.) Our clients that work in regulated environments have very little difficulty in selling the value of testing. However, custom development—at least at this time—still suffers from a bad reputation for quality, so you can combine this value with the previously mentioned one when selling testing.

Reducing risk of loss of whole missions or even lives: If you work on weapons systems, there is significant risk that, if such systems malfunction, so-called friendly-fire incidents or other mishaps could result in the death of the war-fighters trying to use those systems to military advantage. Alternatively, the malfunction could cause enemy warfighters to escape demise and thus live to effect death or injury on friendly troops. Even when the results of failure are not so dramatic, there are plenty of mission-critical systems in banks, insurance companies, investment companies, government agencies, electrical power companies, oil refineries, telecommunications and Internet infrastructure, and other socially critical entities. If these systems fail, the lives of users and other members of society can be significantly affected. It is often—but not always—the case that the necessity of testing of these critical military and nonmilitary systems is obvious to most stakeholders, but the wise test manager is ready to make the case. Be careful not to overplay this value, though, because it’s a fine line between accurately stating the potential consequences of insufficient testing and verging into melodrama.

This list is not necessarily complete. Different stakeholders will have different values that they receive from testing. When you’re communicating about the value of testing, you must take into account the stakeholders’ perceptions of those values. Telling a stakeholder about values they care about will motivate them to respect and support the test team, while talking about values they find irrelevant to their needs will have the opposite effect. You should ask the various test stakeholders about the objectives they have for testing, as discussed in Chapter 1, as part of deciding which of these values to discuss with them.

Let’s look more closely at how to use cost-of-quality analysis to quantify the benefits of testing. These benefits can be quantified either in terms of money saved or time saved. Cost of quality says that, anytime we plan, build, deliver, and support a product, we have costs associated with doing so. If the revenues associated with the product exceed the costs, then we have a profitable product, but we can increase the profitability of a product by reducing the costs. Some of these product costs are unrelated to quality, and cost of quality ignores those costs. Cost of quality says that there are two general categories of costs that relate to quality:

cost of quality = cost of conformance + cost of nonconformance

The cost of conformance is money (or effort) spent to deliver a quality product. There are two categories of cost of conformance:

cost of conformance = cost of prevention + cost of detection

Cost of prevention includes various quality assurance costs, such as training and process improvement, which the organization spends to try to reduce the number of defects introduced into the product. Cost of detection includes those costs of testing that we would spend even if we didn’t find any defects, such as test planning, test analysis, design, and implementation and the first time we execute a test.

The cost of nonconformance is money (or effort) spent on quality problems with the product. There are two categories of cost of nonconformance:

cost of nonconformance = cost of internal failure + cost of external failure

Cost of nonconformance includes both direct costs for defects and indirect process costs (e.g., schedule delays and test downtime due to bad test releases). The defects and their process repercussions result in rework, repairs, delays in schedules, idle time while waiting for resolution of problems, and so forth. If the defects are found and fixed prior to release, these costs are classified under internal failure; if the defects are delivered in the product, these costs are classified under external failure.

When defects are found and fixed prior to release, that’s a good thing, of course. However, the test group must isolate and report the defect. The defect review committee must review and prioritize the defect. If appropriate, the development group must fix the defect and unit-test the fix. The release engineering team must deliver a repaired build. The test group must then repeat one or more tests to confirmation-test the fix as well as run some (or maybe even all) of the previously run tests for regression testing. While this is better than delivering a low-quality product to the customers—for a number of reasons, one of which will be seen in a moment—this effort is classified as rework, which is a form of inefficiency for the software engineering process. Depending on the quality of the incoming software, anywhere from 25 to 75 percent of the test budget is cost of detection, while the remainder is cost of internal failure. Similarly, as much as 75 percent of the development budget can be cost of internal failure.

When the defects escape to production and plague our customers or users, then we have cost of external failure. These expenditures often include half or more of the technical support costs, along with 100 percent of the expenses incurred when creating, testing, and deploying field fixes, product recalls and refunds, liability costs, and lost sales due to quality problems. Worse yet, the average cost of an external failure is always much greater than the cost of internal failure. We have seen the average cost of external failure range from 200 to 5,000 percent higher than the average cost of internal failure, on a per-defect basis.

This means that, expensive as testing can seem when viewed without the context of the alternative of failures in production, it typically saves many, many times more money (or effort) than it costs. This is why, as we wrote in Chapter 1, we can calculate the return on the testing investment as follows:

Test ROI

Cost of quality is a very well-established technique, with over a half-century of published results across many industries. It is a technique you can apply easily because the math behind it is simple.²

As described earlier, you can discuss the value of testing both in static terms and in dynamic terms. Statically, you can capture benefits such as cost of quality and even the qualitative benefits to talk about the value of testing that has been delivered to current and past projects. We refer to this as static in the sense that the decisions have already been made, the actions already taken, the value already delivered. Discussions of static value allow you to promote the test process and team as currently working and constituted.

Dynamically, though, you can talk about values that can be delivered in the future. For example, you can talk about increasing the return on the testing investment by increasing defect detection effectiveness, phase containment, and other potential improvements. Discussions of dynamic value allow you to make forecasts of how improvements to the test process or test team could deliver even more value. Alternatively, you can talk about how proposed decisions in terms of future project or process activities might reduce the value delivered by testing. This ability to forecast, the ability to make solid predictions about what certain decisions mean in terms of testing and quality, deliver more value to the organization.

Let us conclude this discussion on selling the benefits of testing with some general observations. In addition to the needs and objectives of the stakeholders influencing the test benefits they care about, the industry and the organization influence the relative value of these benefits. For example, mass-market software vendors will tend to highly value the benefits of smooth, predictable releases and reputation for quality, while safety-critical system makers tend to highly value the benefits of regulatory compliance and reduced risk of loss of life. Be sure to take these considerations into account when preparing your marketing pitch for your stakeholders.

In addition, remember that your stakeholders are not testers. This means you need to communicate in terms that they can understand and appreciate. Whenever you are trying to convince a stakeholder that some particular testing activity is valuable to them, first ask yourself the question, “What’s in it for them, and how will they see it?” For example, if you are talking to a business analyst manager about the importance of traceable requirements and coverage analysis, don’t say, “We need traceable requirements so we can cover the requirements.” Instead, say, “If we have traceable requirements, we can make sure that our tests cover those requirements. That way, we can help you have confidence about which requirements are satisfied while also letting you know which requirements are not satisfied and what defects exists for those requirements.”

Last, remember that your stakeholders are smart. Just because they don’t understand testing doesn’t mean that they can’t see through an inflated business case or value. When you’re talking to fellow managers about how testing will benefit them, be accurate and, if anything, undersell a bit. As Tom DeMarco wrote, a good manager has a nose for lies and exaggeration.³ At the same time, don’t hedge and waffle, because that will damage your credibility.

The secret here, which is no secret at all but practiced by many successful test managers around the world, is this: In plain terms that they can understand, tell stakeholders how testing will benefit them, make solid commitments about the value of testing, and then keep (or exceed) those commitments. Doing so will allow you to get the resources you need for your current projects and programs. As your credibility grows through promises made and promises kept, you’ll be able to spend that political capital to get more resources for testing, to motivate earlier and deeper involvement for your test group, and to attract support for test process improvements and scope expansion.

This political capital will also help you in the darker hours, when you have to defend the test group under difficult circumstances. That is the topic of our next two sections.

4.2.3Creating a Defensible Team

It’s great when projects go perfectly, exactly according to plan. It does happen, especially with smaller, simpler projects. However, as the size and complexity of projects increase, so does the likelihood of significant unexpected events. From a risk point of view, we can say that the larger and more complex the project, the greater the number of project risks, the greater the likelihood that some of those risks will become outcomes that must be dealt with, and the greater the impact some of those risks can have. Ideally, the project management team would have all of these risks cataloged and managed, but even in that case, serious project consequences can ensue. In the real world, on many projects there are unforeseen risks that suddenly blossom into project crises.⁴

So, the expert test manager will prepare to handle these risks. For the foreseen risks, mitigation and contingency plans for each test-related risk should be part of the test plan. For the unforeseen risks, the test manager and the test group must be prepared to react nimbly. Whenever a risk becomes an outcome, the best managers respond, first and foremost, by taking prompt (but not rash) and efficient (but not penurious) actions that will minimize the consequences that most directly threaten project objectives.

To take the correct action, it’s often important to understand exactly what has happened. For example, if you have a fire in your house, throwing water on that fire might be a good idea, but not if the fire originates from an electrical short or a deep fryer full of burning oil. Similarly, if the project is in trouble because of an enormous defect report backlog, having testers spend more time running tests and finding further failures will probably be less helpful than having testers support the developers in the location and removal of the underlying defects. When a crisis arises, take the advice of Rudyard Kipling: Keep your head when all around you are losing theirs. Objectively, and as calmly as possible, try to determine what has gone wrong so that effective measures may be taken. The focus should be on the effective measures, not on evading culpability. The word excuse shares many letters in common with the word success, but no one mistakes an excuse for success.

Regrettably, when things go wrong, some people will not seek first the actions that must be taken. Instead, they will look for causes to blame. Even worse, this search for causes is often focused on mistakes people have made rather than processes that failed. In some cases, the test process or the test group will be where the failure lies, while in other cases testing will simply be a convenient or simple explanation for why the project is in trouble. As a test manager, if your team is being blamed, you need to defend the test group against unfair accusations while at the same time forthrightly and honestly holding yourself accountable for your contribution to the problems that have occurred.

As the saying goes, the best defense is a good offense. In this situation, that aphorism means that you should act, before crises happen, in a way that helps avoid the crises and, when crises do happen, respond effectively to them. Here are three up-front actions you can take, before the project starts and throughout the project, to make your team less likely to be blamed for project disasters.

Open communication: In Chapter 2, we gave our opinion about why open communication within the test team is helpful in terms of building a strong, loyal team. Open communication within the team also helps when the project gets into trouble, because matters are clearer to all participants in testing about what is going on, why it happened, and what is going to happen next. Open communication with managers outside the test team has similar effects. If external project and product stakeholders and participants have been kept informed throughout the project, they are less likely to make false assumptions about the culpability of the test team and test process.

Open communication also means that, after the project gets into trouble, you, as the test manager, admit your group’s role in the problem, if any. At the same time, resist any temptation to blame particular people within your group. As the manager, you are accountable for your group; it is the mark of the cowardly manager to attempt to shift blame onto those who work in their group. To the extent that you need to coach or, if necessary, discipline someone within your team for the problem, it should never, ever be done in public. Further, open communication means that, whether the test group is responsible for the problem or not, you should communicate with your fellow managers and stakeholders about how the test group intends to help the project recover.

Good documentation: It’s not possible for people to remember exactly how the project got into trouble. Eyewitness testimony, often relied upon in trials, is in fact one of the least reliable forms of evidence. People will forget key facts, or may simply never have been aware of them, no matter how much you strive for open communication. In the regrettable case that you are called upon to prove that your group should not be blamed, good documentation can be priceless. For example, email trails that show how decisions were made about testing by groups of stakeholders rather than by the test group alone can explain why things were done. Test logs that show how external factors affecting testing can provide a similar role. There is a fine line between documenting what happened and working proactively to have ready excuses, though, so be careful here.

Strong processes: When there are clear, accepted test processes, then the reason why events transpired in a particular way will be more clear to people. Making sure stakeholders understand why and how the test group works the way it does is not just a matter of defending against blame. It’s also about building consensus about the right way to go about testing in a particular situation. When this consensus exists, it’s much harder for people to credibly second-guess what testing did. Instead, this shared understanding of the test process builds confidence in the test group and helps the test manager manage stakeholder expectations.

Open communication, good documentation, and strong processes should all work to effect a broad understanding, across the organization as a whole and within the project stakeholder team, of what the role of the test group is, why it does what it does, how it manages project risks, and how it responds to project crises. This point goes back to topics covered in Chapters 1 and 2: Establish clear expectations about the objectives and responsibilities of testing and what testing will do within any given project. The definition of roles and responsibilities discussed in Chapter 2 helps make this clear. A clear test strategy, as discussed in Chapter 1, and good test plans, as discussed in the Foundation and Advanced syllabi, help make clear the approach followed and the reasons for it.

You might be thinking, “Okay, I’ve fostered open communication in my test team, but I’m not sure how to go about putting in place good documentation and strong processes.” Fortunately, you don’t need to create these from scratch. The ISTQB Fundamental Test Process, from the Foundation and Advanced syllabi, can give you a good starting point for process, and those syllabi also contain a number of ideas about how to document properly and incorporate good feedback mechanisms.

Good documentation and open communication have implications in terms of test status reporting. In Chapter 6, we’ll discuss the importance of good test results reporting dashboards. Such dashboards provide appropriate, timely, credible, and useful visibility into not only the status of the product, but also the workings of the test process. Status reports, test execution progress, coverage, and defect information should provide stakeholders with what they need to make smart decisions and to see trouble coming in the project when it does. To some extent, if stakeholders are surprised by product quality problems or test process breakdowns that have been brewing for days or weeks, it is quite likely that you need to revisit your test results reporting practices.

We should point out that, like everything else in testing, context matters. The product, the project, the organizational structure, and the software development lifecycle influence the way and degree to which a test team needs defending. For example, in Agile projects, it is often the case that one or more testers are embedded in each team. This close collaboration with the developers and with the business can help to prevent scapegoating of the testers when a sprint or even an entire project goes awry, though testers may bear the brunt of Agile disasters in some cases. The solution here is to ensure that the Agile principles of efficient, effective communication; close collaboration; and good teamwork replace any effort to blame one or another participant for what’s gone wrong.

4.2.4Protecting and Supporting the Team

Nobody likes to be told how to do their job. Winston Churchill said, “I am always ready to learn, but I don’t always enjoy being taught.” However, as a test manager, you have probably noticed the plethora of people who are happy—or sometimes unhappy—to interfere with your test group and your management of it. In some cases the intention is good (as when the instant expert on testing wants to dictate what actions they would like to see your group take), and in other cases the intention is bad (as when the individual wants you to simply get the heck out of the way). Either way, any situation in which a nontester is telling you or your group, which should be composed of professional testers, how to do your job, is a problem sign.

By the way, we want to clearly distinguish between unreasonable interference and reasonable interactions with a service organization. The former occurs when someone tells you how to do the job, insults the way you do the job, or creates obstacles to doing the job, while the latter occurs when someone tells you what they need from you. If it wasn’t obvious from Chapter 1 and earlier in this chapter, we consider the testing group to be a service organization in most circumstances. Talking to stakeholders about what objectives they want you to achieve and the benefits they need from your group is a conversation you should initiate yourself, and if it’s initiated by one of your stakeholders, you should welcome it. Having stakeholders telling you how to achieve those objectives, or telling you that the objectives are being incompetently achieved, or telling you that you are in the way, well, these things indicate some sort of problem.

What’s the Problem?

This is one of the obvious questions: Okay, it’s a problem, but is it your problem or theirs? There are two answers to this question. One answer is that it could be either your problem or theirs. You need to understand why the problem is happening, so that you can address it. The second answer is, either way, it is your problem. If one of your stakeholders feels that your group is not doing their work properly and feels strongly enough about that to behave in an aggressive or socially inept manner, then there’s at least one dissatisfied testing stakeholder out there. So, let’s start with the question of why the interference is happening, and then we can address ways to deal with it.

Misunderstood and underestimated: It’s often the case that stakeholders outside of the test team do not see that the test process includes activities before and after test execution. “Why do you need to see the requirements specification,” the business analyst may ask, “since you won’t need to test that function for a few months?” This is your opportunity to explain the value of early testing. Make sure, when you do so, to explain how early testing benefits the business analysis team. Moving beyond this specific example, you, as an expert test manager, need to be able to explain the entire test process and how that process benefits the organization.

Undervalued testing: If you don’t do a good job of marketing and selling testing, as discussed earlier, stakeholders might (and often do) see your group as a low-value-add. That’s likely to result in interference with your group whenever your group’s behavior is anything other than obsequious. Some test groups respond to this situation by getting their backs up and asking for management “to support them in enforcing” the test process, test entry or exit criteria, or other similar rules. This is referred to this as the “process cop” model of testing, which leads to the problem of “testing seen as obstructing progress” discussed later. Instead, refer to the material discussed earlier and in Chapter 1 about how to make sure your group delivers value to your stakeholders and your stakeholders see that.

Undervalued testers: There is a subtle difference between this item and the preceding one. In undervalued testing, the stakeholder belief is that testing itself is of limited value. In undervalued testers, the stakeholder belief is that testing itself is valuable but the current team is not very valuable. Alternatively, the belief can be that testing is valuable but it does not require any special skills, so anyone can test. If you are managing the skills of your team as discussed in Chapter 2, you should be able to demonstrate, in great detail, the skills required from your testers, how they deliver those skills, and the wide extent of the skills required.

Right tester, wrong place or right testing services, wrong test group: The impression here is that someone in the testing group would be better in another role or that your approach to organizing the test group is wrong (e.g., recall the specialized versus generalized discussion in Chapter 2). Okay, this is a tricky one. There were probably good reasons why you assigned people to different roles and why you organized the group the way you did. That said, such reasons are always trade-offs. Be ready to reevaluate the trade-offs. If the stakeholder is simply suboptimizing (i.e., valuing their own needs over the wider needs of the project, product, process, or organization), then you have made the right decision. However, be open to the possibility that you can do better in this regard. It’s one of the hardest trade-offs to get right as a test manager, so if someone tells you that you are doing it wrong, listen and think. Reflecting on the test groups we have set up, we are certain that there are many decisions about role assignment and organization we would do somewhat differently if we had a chance to do those over again.

Ineffectively utilized testers: This is a broader variant of the problem mentioned earlier—not only one or two testers are misplaced, but they’re all misplaced. Or more exactly, their roles are not well defined. The assertion here is that your team organization model needs to be rethought completely. As earlier, this can be a matter of the individual seeing their own tree and not the whole forest, but it could be that circumstances have changed and your organization should as well.

Testing seen as obstructing progress: This is a very dangerous situation, and one that should be addressed immediately if it comes up. Developers will become frustrated with the testing obstructions and start agitating to have testing responsibility put in the development team. On more than one occasion, Rex has seen entire test groups disbanded, test managers fired, and testing outsourced once this impression became widely held, in some cases even when the test team was enforcing processes at the request of executive management. If you are trying to rigidly enforce entry and exit criteria, trying to force developers to document unit tests and unit test defects, trying to block releases, and otherwise trying to put the brakes on the project in the name of more testing and higher quality, you are in danger of building this impression. In some organizations, nontest stakeholders want and value process cops and quality cops who play such a role, but usually it’s a losing game. Instead of chasing this path, refer back to the discussion in Chapter 1 about how to identify the proper objectives for testing from your stakeholders.

Test management seen as obstructing progress: This is a special case of the preceding problem, where the test team in general is seen as a positive force in the project but one or more of the managers are seen as obstructive, uncooperative, inefficient, or just plain unpleasant to deal with. Frankly, we do have a problem, as professional testers, in that some people seem to have come to the conclusion that being personally obnoxious is part of having integrity and independence, and that reputation has stuck to our profession to an unfortunate degree. In our experience, a leading indicator of the unsuccessful test manager is someone who feels that being a test manager is a license to tell everyone on a project exactly how they feel about the quality of the software, the quality of their work, and how everyone should be doing their work. Again, going back to the ideas mentioned in Chapter 1, the successful test manager is usually one that focuses on how to provide useful testing services to project, product, and program stakeholders.

Role clarity problems: In the absence of a clearly defined set of responsibilities, objectives, and goals, the role of testing is often very fuzzy, especially for stakeholders who’ve never been involved in testing. The best practice, again as discussed in Chapter 1, is to have a clearly defined test policy that makes these things plain. If you already have such a policy in place and you are still finding that people don’t understand testing, then the problem is one of sales and marketing for the test role, as discussed earlier in this chapter.

The wise test manager works proactively to address these underlying preconditions of interference. If such a condition comes to the test manager’s attention, in spite of efforts to address them in advance, then the test manager moves quickly to resolve the problem. When these conditions persist, they will often lead to various resource and respect issues that can be very dangerous to the test team and at the very least undermine your attempts to sell and market testing.

Let’s look at some of these resource and respect issues that can arise in these circumstances.

External Interference

On big holidays like Thanksgiving, at big family gatherings, or at occasional parties, Rex is often the cook. He enjoys it, in part because it’s like a mini-management challenge. There’s planning, preparation, execution, and closure (the meal), all within a few hours. The few times he hasn’t enjoyed doing it came when people decided to hang around in the kitchen while he cooked to tell him what to do. He doesn’t mind people hanging around and helping, of course, but telling someone what they’re supposed to do next is usually not helpful, and neither is preemptively taking over some tasks already underway without someone asking for help.

It’s irritating when someone decides to interfere with something you are managing. This sometimes happens in testing, when other project or product stakeholders decide to try to manage the testing work or even take some of it over. For example, if the development team feels that testing has become a bottleneck for release, they might try to dictate a reduction in testing scope based on what they feel should be covered and what should be left out. While many test strategies, including risk-based testing, do involve developers in discussions about test scope, only directed test strategies allow outside parties to completely determine it. If you are not following a directed test strategy, then such input would constitute unwarranted external interference.

While trying to change scope or curtail the testing schedule is perhaps the most common form of external interference, it’s not the only form. Outside stakeholders might try to determine the way in which test cases will be formatted and documented, especially if they are involved in reviewing those cases. Outside stakeholders might offer unsolicited advice or even inputs to your test automation programs. They might also provide unsolicited critiques or even modifications to templates, test documentation standards, and other testing work products.

As discussed, the reasons for this kind of interference can vary considerably. You need to understand why this is happening, if it does. You also need to be receptive to input from project stakeholders and interfacing organizations because a brusque response to these inputs can trigger (or reinforce) the “testing/test managers are obstructive” problem mentioned earlier. However, the test organization is supposed to focus on quality, and this external interference is usually not focused on quality. So you need to politely but firmly maintain a focus on your mission.

If external interference happens, remember that it is a symptom of a larger problem. A bad manager just reacts to symptoms, thrashing inefficiently in the face of crises like a badly operated marionette. A good manager resolves the underlying problems and thus spends a lot less time reacting to symptoms. Remember that, in cost of quality terms, any time spent reacting to a problem that has created a crisis is a cost of internal failure—and thus a form of inefficiency.

Micromanagement

Micromanagement is similar but more severe than external interference. External interference usually happens only when outside stakeholders come to a conclusion that they need to direct testing so that it better serves their objectives or institute constant monitoring of the test work being done so that they can stop anything they don’t like. Micromanagement usually happens when a testing stakeholder—usually one in a senior or peer management position—has lost trust in the test management team’s ability to do their job. This can be because the test manager and the team have not done a good job in selling and marketing the mission and strategy of testing, or perhaps the mission and strategy are not clearly defined to begin with.

Here are some classic symptoms of micromanagement:

• The test team is the only team required to account for their hours, perhaps in hourly or even 15-minute increments, while the other teams are not.
• The test results are closely scrutinized for inconsistencies and minor flaws, with more attention given to the mistakes of the test team than to the meaning of their results.
• Members of other teams demand and receive the right to change the status of tests and defects and use this capability frequently to modify the test results.
• Managers of other teams demand and receive the right to redirect test resources, including testers, to assist with nontest tasks, perhaps without any need to consult or even inform the test manager.

If you see symptoms of micromanagement—or even attempts at micromanagement—you need to act. While you might be tempted to react by saying, “Oh, well, that’s just Joe [or whoever’s being the micromanager]; it’ll be easier to just go along with this nonsense than to fight it,” the chances are good that, in the long run, it won’t be easier. If you have all the responsibility and none of the authority, how long do you suppose it will be before someone else’s bad decision comes around to nail you and your team?

When we say that you must act, we don’t mean that you should simply put your foot down and refuse to allow it. First, you need to understand why the problem is occurring. If multiple parties are involved in micromanagement, keep in mind that each party might have its own reasons for doing so, even though the behavior is the same.

In addition to the reasons previously discussed, there are two other reasons micromanagement occurs that we have seen in our careers. One cause is when people get appointed to a management role but they simply cannot get out of the habit of being in a lead role as a subject-matter expert. A subject-matter expert or technical lead is supposed to advise their colleagues on how to do some elements of their work, and some leads (not the good ones) get a real thrill from being the smartest guy (or woman) in the classroom rather than being an effective teacher, if you follow the analogy. Once promoted to management, they simply can’t get out of the habit or in fact don’t even understand the difference between a lead and a manager. If a former test manager or director with these kinds of predilections is appointed to a project or program management role, they might feel an overwhelming temptation to micromanage their test successor.

This is a tricky problem to handle because you are dealing with what could best be called a mild personality disorder (keep in mind that none of us is a psychologist, so we are not diagnosing here). Our practice with these kinds of managers is simply to avoid giving them opportunities to micromanage. We look for every possible way to ensure that, by the time they become aware of a problem, our teams have already solved it or are well on the way to solving it. By working “around” the micromanager, engaging with other colleagues and fellow managers, you can often minimize the impact. You are probably not the only person to have developed a healthy dislike for this individual; no one likes a smarty-pants.

Another reason for micromanagement is when a manager has had a long, unpleasant experience of managing incompetent or dishonest employees. This can lead to a habit of having to micromanage employees because otherwise nothing (or at least nothing good) gets done. This is actually a simpler problem to solve because it’s merely a matter of winning trust through constant, stunning excellence on the part of you and your team. Notice that we said “simpler problem to solve,” not “easy problem to solve.” It’s not easy to be really good all the time, but if you can raise your game to that level, you can earn your way out of the micromanager’s basilisk glare.

As with external interference, the first step is to understand why the micromanagement is happening, and the next step is to address the cause(s). In addition, you should work to reduce contributing factors through the following actions:

• Get involved in the overall planning and estimation for the project or program. If you don’t know how to participate effectively in such exercises, then you need to sharpen your project management skills (see Chapter 5).
• Make sure that you are clear in your communications during these planning and estimation activities. Explain the test strategy, the test plan, and the estimate to the stakeholders. Have reviews and sign-offs, yes, but also make sure people really understand why you intend to do things and how those things will benefit the organization.
• As project work continues, continue that habit of clear communication about your progress. Keep people apprised of where you and your team are. If things go badly, develop a good way to solve the problem, and then communicate the solution. (Our experience is that simply coming to managers with a problem, without offering any solution, is a great way to trigger any latent micromanager tendencies among those managers.)
• When you are asked to provide additional information about test results, test status, test progress, or test activities, respond forthrightly and without delay. If a request for information is based on an honest desire to better understand, then you risk alienating a friendly stakeholder with irresponsiveness. If a request is a symptom of micromanagement, you will accomplish very little by resisting the request directly because it simply feeds the narrative that you and your team have something to hide.

In general, micromanagerial tendencies will be exacerbated or perhaps even triggered by feelings of uncertainty about testing work and surprise at the testing results. Working to increase transparency and understanding of what the test team is doing, is planning to do, and might have to do should certain project risks eventuate can help to reduce the likelihood of micromanagement.

Three final notes about micromanagement: First, if the micromanager is your boss, be very careful about how you handle the situation. Every manager who succeeds over the long term learns how to work successfully with their managers, accommodating their work habits and personalities; we cannot think of an exception to this rule. Second, remember that micromanagement is not a problem because it offends your pride. Micromanagement is a problem because outside micromanagement by underinformed individuals cannot be as effective as competent management by competent test managers. Third, don’t confuse situations that have offended your pride with micromanagement. If you make a mistake and your manager gives you sharp and clear direction about how to avoid that mistake in the future, that’s not micromanagement, it’s intervention. If you don’t like the way the intervention was delivered, please refer back to our first note, at the beginning of this paragraph.

Disrespect

For years now, Rex has been writing about the “second-class citizen” problem with testing, and he wasn’t the first to do so.⁵ We’d love to be able say that this problem was solved, but unfortunately it’s not. We still see situations with clients where the test team is seen as inferior and treated with disrespect.

As a manager of any team, including a test team, you cannot tolerate such a situation. It’s bad enough for people outside a team to direct disrespect at that team, but should the manager acquiesce, any remnants of morale will be destroyed. What signs of disrespect should you watch for? While not a comprehensive list, here’s a start:

• Sarcastic, unfriendly comments about the test team’s capabilities, the skills of the testers, or their value. Just as every dog knows the difference between being tripped over and being kicked, you should know the difference between this and good-natured repartee.
• Openly insulting comments. Here, no subtlety is involved. If someone says something like, “Another typical stupid status report from the test team,” right in the middle of a project meeting, guess what? You’re disrespected.
• Working assignments that have nothing to do with testing. While this can be a form of micromanagement, a failure to understand what testing does, repeated actions in this regard in spite of attempts to resolve the confusion, will often be a sign of disrespect.
• Insufficient time and resources for testing. Again, this could be a failure to understand what testing does, but it could also be a way of saying, “We don’t think you’re competent to manage this effort, so we’re going to give you limited resources.”
• Insufficient training. This can be a symptom of the “how hard can be it, just make sure it works” opinion about testing. However, it can also be a way of saying, “Those testers are hopeless; no amount of training could make them valuable team members.”
• Information hiding or barriers to information access. Usually these situations come down to a severe lack of trust or an opinion that you and your team can’t handle information because you’d make bad decisions with it.

When we’ve encountered respect issues, the typical (but not universal) cause is a failure to understand the value of testing and the ways the test team contributes to success. As with the other situations mentioned, know the cause first, then address the cause.

A special case in this category is disrespect due to sexism or racism. When test teams comprise disproportionately people of one gender—and I’m sorry to say we’ve only seen or heard of cases where the gender was female—then sometimes the disrespect is gender bias. When testing is outsourced, we’ve seen situations where racial elements enter into the disrespect. With that nasty possibility acknowledged, we would encourage you to look for every other way to deal with the problem of disrespect before attributing it to sexism or racism. Certainly, if you hear sexist, racist, or other comments that create a hostile work environment directed at your team, you have an absolute duty to act, immediately, both as a human being and as someone with a responsibility for protecting the interests of your employer. However, if you call out to management, “Help, help, we’re being oppressed,” in response to every slight, you will damage your team’s credibility.

Reorganization and Resource Reallocation

Finally, there is one last topic to address with respect to defending the test team. In the absence of clearly defined testing tasks and testing value, reorganizations and reallocation of test resources can lead to significant reductions in the testing resources available. We have seen this involve test hardware, test networks, and testers. These situations often affect test teams disproportionately. We have seen test teams cut by 50 percent when their development colleagues lost only 10 percent of their staff. Worse yet, in some cases after the resources were withdrawn, the expectation was that the committed testing work would nonetheless be done.

This situation can occur when the rest of the organization doesn’t understand the testing tasks or they underestimate the work to be done. A typical exacerbating factor is that people don’t understand the value of testing, especially in situations where the test manager has not bothered to measure the value. Remember, the value of development is clear—no developers means no software—while the value of testing, both quantified and qualitative, is less obvious. This is another reason why defining and communicating the value of testing, as mentioned earlier, is so important.

Note that this problem is not the same as outright disrespect, because the underlying message is not “testing has no value.” Typically, the underlying message is “testing has value, but we’re not sure how much of that value we need.” In the absence of clearly defined value for testing, half of the value is just as good as all of the value. However, going back to the dog’s perception of the difference between being tripped over and being kicked, the understanding of a lack of malice must be cold comfort to the dog with a bruise on its back. Similarly, if half of your test team is fired, you are unlikely to be mollified by an executive who says, “We really respect the valuable testing work your team does, but we have to focus resources on development and other high-ROI activities.” The worst part of that situation could be realizing that better communication about the value of testing might have prevented it.

4.3Placement of the Test Team

We have worked with test teams around the world, within various organizations from small to large, in a wide variety of industries. There is considerable variation in organizational structure. There is also considerable variation in how test teams are organized and how those teams relate to the organization as a whole.

We’ve also seen situations where the structure and relationships of the test team changed for each project, and often in those situations the larger organization was restructured to suit the needs of each project. For example, a testing services provider will need to form teams for a client’s specific project or program. Mass-market and enterprise software development organizations often form special teams to work on particular releases.

In some cases, the structure and relationships with the wider organization work well, and in other cases, the structure and relationships are a source of friction. The same approach that works in one situation may fail in another. A given structure may be perfect for a specific project, program, or organization, but there is no single structure that is perfect for all projects, programs, and organizations.

As is the case elsewhere in testing, though, just because context matters doesn’t mean there are no best practices. We have observed many common characteristics in testing teams where the structure and relationships with the wider organization worked well. In the following sections, we’ll examine some of those characteristics.

4.3.1 Independence and Reporting Structure

Most testing exists within the wider context of some endeavor such as a project, program, or operation to deliver or utilize some set of features, and usually there are schedule and budget targets for that endeavor. It’s seldom realistic for test teams to expect that those targets must be subservient to quality targets. Perfect software is not possible within the constraints and realities of most systems and organizations. The proper balance of schedule, budget, features, and quality will vary from system to system and from organization to organization.

The test team’s role in achieving that balance is to focus on quality. This usually does not mean being a quality cop, someone who enforces a certain level of quality, but rather it means assessing the quality of the product or system. That assessment must be accurate, timely, and credible. It then must be reported honestly. For these things to happen, it is necessary that the test group receive adequate resources and time. It is also necessary that the testers and test managers be able to speak the truth, as they understand it, about the quality of the software without having to worry about negative consequences for themselves or the group.

This privilege—indeed, an essential duty for testers—to speak the truth is often accomplished by independent test groups. Independence means that the reporting chain of the testers leads to senior or executive management rather than to product, project, program, or operational managers. The reasoning is that if testers report to managers that are incented primarily by schedule, budget, and feature targets, those managers will censor the testing assessment before passing it along, and testers will learn to self-censor. If testers report to managers who have the broader perspective of the organization in mind, there will be no incentive for the testers to withhold the facts, and those managers will be able to make a balanced judgment.

For example, one of our clients has a structure where system-integration testing is performed by a separate organization, which provides services to the various programs that are underway. Each program has an assigned test manager. The test managers report to a director of testing. He in turn reports to a vice president in IT, whose other direct reports are not program managers or their directors. The vice president reports to the CIO.

As another example, one of our clients is following an Agile approach to software development. Most of their testers are assigned, with a dotted-line reporting structure, to work within Agile teams. (Other testers who are working on support activities such as test automation frameworks are in separate teams, not within the Agile teams.) However, they report directly to test managers, based on product line. Those test managers report to a director of testing, and he reports to the CIO.

Given a proper reporting structure, the test group should deliver honest, unfiltered assessments of quality, with no fear of retribution. The test management team must ensure that the reporting structure is such. However, the test managers must also remember that all communication is a two-way street. This duty to deliver a frank assessment must not be taken as a license to offend or obstruct, though. Not only will such a communication style produce significant long-term damage in many cases, it will also interfere with the receipt of the message by the very people you are trying to inform. The entire test team must instill and practice a style of communication that is honest and at the same time respectful, relevant, and helpful.

4.3.2 Access to Information

Independence of the test team is important, but it can be in tension with another important issue related to organizational placement of the test team: access to information. Some organizations that go too far in insulating their test teams find that those test teams can’t get the information they need to do their testing. We’ve seen this happen with certain outsource testing services providers as well.

What information do the testers need? In short, they need information about the product, project, and process. Product information includes requirements and design specifications, among other test bases and test oracles. Project information includes project plans and project status reports. Process information includes defect trends, defect clusters, and other process capability indicators.

Not all of the information comes in the form of documents. Informal communication such as email, hallway and lunchroom conversations, whiteboard sessions, and phone calls are often essential channels of information for the tester. This is another reason testers must communicate in a way that is respectful, relevant, and helpful. By doing so, testers encourage bidirectional communication and are much more likely to receive the information they need.

We’ve seen examples where the test managers did not work to ensure this kind of good, mutually supportive communication, with disastrous results. Testers were given minimal information and left out of the information conversations entirely. When the test managers complained that they didn’t have the information needed to do their job, the development managers countered that the entire test group was merely demonstrating its irrelevance and inability to operate within the context of the organization. In one case, the end result was the dissolution of the test group and outsourcing of all the testing.

4.3.3 Skills

We discussed skills for testers extensively in Chapter 2. It’s an important topic and has various effects on the placement of the test group and the way it operates. A group with the perfect skills for one situation may be exactly wrong in another. An individual tester or test leader with the right or wrong skills can make a difference in the way the entire test group is perceived. These are two more reasons the task analysis, skills inventory, and skills management discussed in Chapter 2 are so critical.

Rex’s company has clients that work in highly complex business domains, such as oil and gas exploration and industrial control systems. As the director of testing explained to Rex when he was asked why he hired only domain experts for his test team, “I can teach them what they need to know about technology and testing. I hire smart geologists and petroleum engineers, people with experience in the oil and gas industry. Those people can learn technology and testing. What I can’t teach someone without domain expertise quickly enough is how the system works, the problems it actually solves. The learning curve is simply too long, and by the time those people are useful testers, they’ll have made too many mistakes that have damaged the test team’s credibility.”

Rex has clients and vendors that work with his clients who really need skilled testers. For example, if you are managing a test group that provides outsourced testing services to your clients, your company is selling expertise in testing, plain and simple. The ability to bring that expertise to bear for your clients is paramount. If you are hiring a testing services provider to help your team, a gap in testing expertise is often what you need to fill. If you are a director of testing, as some of Rex’s clients are, and using outsourced testing services providers, you need to focus on the testing skills those service providers bring.

He has yet other clients who work with extremely advanced, leading-edge technologies. These companies often see technology as a key competitive advantage, and their focus is on using that advantage to win in highly competitive markets. For example, he has clients that make and sell enterprise software, providing or enabling the benefits of cloud computing, virtualization, and so forth. Testers in those situations must have deep knowledge of system configuration, networking, and similar technical skills.

Be aware of the fact that the proper skills mix might not be uniform across the organization. In some cases, the specific needs on a particular program or project could differ significantly from one program or project to another. This is especially important if you are the director or vice president of testing for a large organization, providing services to various software development, maintenance, and operational groups that build, maintain, and support disparate software products. We have seen this situation even in smaller organizations, where different groups are involved with very different kinds of products or following different software lifecycles.

For example, some of Rex’s clients use both Agile and sequential lifecycles, usually for very good reasons. Some of their products are best produced in the quick, iterative manner Agile delivers, while other products require longer lifecycles. For their Agile testers, these people work embedded within development teams, as described earlier. Because of the close work within development teams, stronger technical skills are often required. These testers often need a broader level of experience, too, because the Agile teams are supposed to operate in a self-organizing fashion.

Finally, don’t expect that this consideration of skills is a “once and done” activity. The skills management approach described in Chapter 2 could be misconstrued as meaning that, having determined the critical skills for your group, you simply continue on a linear path toward that ideal group skill level. Nope, it doesn’t work that way. The requirements of the job will change as the technologies, testing techniques, and business domains themselves evolve.

For example, Rex has a client in the retail space. Its IT organization builds software and systems that support everything from the retail sales operations on the store floor all the way to the analysis of big data. Technological advances, and their applicability to retail businesses, have absolutely rocked that business to its foundations in the last ten years, and show every sign of continuing to do so. If the organization’s director of testing was continuing to build the right test group for retail IT in the year 2000, he would be building a hopelessly outdated group at this point.

Internal and external stakeholder expectations will evolve too. For example, if you are the incoming director of testing for a group that has underachieved for years, the stakeholders will be thrilled to see you work through the process, described in Chapter 1, of aligning the testing with their needs. They will be thrilled to see you align the test group’s skills with those required to execute the strategy, as described in Chapter 2. Remember to keep them thrilled. Keep asking yourself, and asking the stakeholders, “What’s next for us as a testing group? How can we continue to get better?”

We have mentioned before in this book that test teams should have a service-oriented attitude and work to provide good services to their stakeholders. You simply must have the right skills to be able to make good on this promise. Both the attitude and the skills are essential. If you and your group apply the right skills and attitude to the mission, objectives, and strategies that fit your organization, you are highly likely to succeed. Testing stakeholders will see your group as the right people to do the job. That means that you and your group are highly likely to attract the respect, receive the information, and have access to the resources necessary to carry out the mission, achieve the objectives, and execute the strategies.

4.4Stakeholder Communication

So, you have a service-oriented test group that has access to the information it needs and the skills necessary to execute the test strategy. Good enough, right? Well, there’s another important element here. Since a key objective for testing is to be a credible, understandable, timely, accurate source of information, it’s also essential to deliver the information stakeholders need when they need it. We’ll discuss specific types of information you can deliver in Chapters 6 and 8, but because we are talking about how test groups relate to stakeholders in this chapter, let’s look at the issue of communicating that information to stakeholders here.

Here are some types of stakeholder information communications that can be useful and for each, the questions that good communication can answer.

• The testing project: Stakeholders, especially project stakeholders such as development managers and project managers, often need to understand where the test team stands in terms of testing. Are the test cases ready? Is the test environment ready? How many of the tests have been run? What requirements, risks, configurations, and other essential test basis elements have been covered? How many defects have been found? Where do we stand on the exit criteria? The answers to these questions help project stakeholders better understand the state of the project.
• Trending information: In addition to understanding the state of the project, stakeholders, especially project stakeholders but also product stakeholders (who include program managers, product managers, and business analysts, among others), need to understand the trajectory of the project and the product. Will we complete the tests, achieve adequate coverage, and satisfy the exit criteria on schedule? If not, how far off will we be and what can be done to get closer to done by the scheduled date? The answers to these questions help stakeholders see possible paths to the most successful outcomes for the project.
• Specific test status and defect status: Project stakeholders and product stakeholders, including individual contributors such as developers and users, often need to understand what works, what doesn’t work, and what is not yet known. Which tests have been run, and which have passed and which have failed? Which defects have been found, which remain to be fixed, and which are resolved? What test-basis elements are associated with these tests and defects? What specific risks and functional limitations are inherent in the current test status? The answers to these questions help stakeholders understand the current quality of the product and make smart decisions about how to manage the quality of the product.
• Test process effectiveness and efficiency indicators: Process stakeholders who are accountable for testing—or who have an interest in knowing how well it is working—often need to understand the capability of the test process. What percentage of defects does testing find? What are the current costs of quality? What percentage of the test basis is covered by testing, and what is the average cost per test-basis element covered? This information can also be useful for project and program estimation.

This is not a comprehensive list, and other metrics will be covered and illustrated in detail in Chapters 6 and 8. The main point of this list is to see some examples of the relationship between particular stakeholders, types of information available from testing, and how that information satisfies stakeholder needs.

Testing information is often ephemeral in its value, especially project and product information. Information that is old may at best be useless and at worst lead to bad decision-making. So good information communication to stakeholders requires working with those stakeholders to ensure that timely, regular delivery of information occurs. This is necessary to support smart, effective decisions. Any test information that is greeted with a chorus of “Oh, cripes, if we’d known that we would have done something differently two weeks ago” was delivered too late.

Test process information is often measured in longer intervals and has more lasting value, at least if the test process is stable. Delivering daily reports on defect detection effectiveness, for example, is likely to swamp the recipient in a sea of transient noise. The signal emerges in the longer term. Even so, establishing timetables for communicating process information is also important.

Just as with the speed of sound, there is an upper limit on the speed and precision with which certain types of information can be generated. Delivering inaccurate information quickly is just as bad as delivering accurate information too late. Part of the job of working effectively across the organization is working with stakeholders to identify their information needs, how testing can support those needs, when that information can be provided, and how to balance speed and accuracy.

This is, of course, a two-way communication. Just as stakeholders need information from the test group, the testers need information from other stakeholder groups. These groups have the same challenges in terms of speed and accuracy. The volume and mode of information delivery is another part of this puzzle. What information do people need, how should it be delivered, and when? Successful test managers work effectively with their colleagues to find the right answer to this question.

We really should say, though, that they work effectively together to find the right answers to these questions, because the answers to the “how” and “when” questions depend on what information is being delivered. An urgent problem that has the potential to delay or even derail a project must be communicated quickly, even if the information might be incomplete. Routine information, such as a weekly status report about testing that shows normal progress, can be communicated in a regularly scheduled meeting or even via an intranet page. As you can see from these examples, the answer to the “when to communicate” question depends on what’s being communicated.

A similar situation exists with the “how to communicate” question. If you publish the same set of test project reports on a weekly basis and everyone knows how to read and interpret those reports, then successful communication can occur via email, intranet pages, and other asynchronous channels. However, if you have found a problem, and performed some analysis to understand the cause of that problem, simply sending a couple of trend charts and a scatterplot or Pareto chart via email will probably not lead to clear understanding by the recipients. A meeting, online presentation, or phone call allows synchronous communication—yes, a conversation—where the test manager can explain the problem and why they think they have found an explanation for it. This is especially critical when the explanation might be politically explosive, such as blocking issues found during system testing that could have been detected during unit testing.

Here’s another important point about effective communication across the organization. There’s an aphorism in public relations that says when interviewed, the person should answer the question they wanted to be asked, not the question they were asked. If you watch reporters interview astute manipulators of public opinion, you will often see this aphorism in action.

This aphorism absolutely does not apply to test managers. Smart stakeholders will see through any attempt to manipulate them, and they will not appreciate it. This attempt will, at least in the long run and often in the short run, damage your credibility. Earlier in this chapter we wrote about the testers, duty to report honestly, and that duty is not served by trying to manipulate stakeholders with selective and biased information communication.

Instead, answer the questions that the stakeholders need answered. Each piece of information gathered, interpreted, and reported should add value for the recipients of that information. Unnecessary or irrelevant information, even if accurate, is frustrating to the recipients and often distracts them from the information they really need, even if you do ultimately deliver that information as well. You are not only wasting the stakeholders’ time, you are wasting your own time and, as we said, draining your precious reservoir of credibility.

Now, when we say that you should provide information that stakeholders need, that’s not the same as saying you should provide information that stakeholders want to hear. Often, test managers are in the difficult position of providing information that is not pleasing to the recipient. Understand what you are trying to accomplish with the communication and what kind of management decision, support, or commitment you are trying to achieve. If you are explaining a problem and a path to solving that problem, be clear about the costs of the problem and the solution, the benefits of the solution, and what will be required to implement the solution. And above all else, be objective.

At the risk of dating ourselves by teasing meanings out of old popular music from the late 1960s, we’re reminded of a refrain that goes, “I’m just a soul whose intentions are good/Oh Lord, please don’t let me be misunderstood.” If your intentions are good—a desire to effectively meet the information objectives for testing—you will be pointed in the right direction when you communicate with stakeholders. However, when you communicate, not only must your intentions be good, you also need to ensure that the message you intended to send was the message that the recipient received and understood.

In some cases, it’s not enough for the recipient to receive and understand your information. If you are trying to propose a particular course of action, effective communication also means that the proposed action is understood. Now, the recipient might or might not act in the way you propose. However, if you are an effective communicator, the recipient will at least understand what you want to see happen, why you want that action taken, and your reasoning behind that proposal.

4.5Creating and Building Relationships

When Rex does assessments for clients, he talks to a lot of people, both inside and outside the testing group. In the opening moments of each interview, he tries to engage in a friendly exchange, where he breaks the ice between the interviewee and himself. Not only is it more pleasant to have a friendly conversation than a tense one, but people are more open and honest with someone with whom they have some kind of positive relationship, compared to a complete stranger—or someone they see as hostile, cold, or inscrutable. Most of the time, Rex succeeds, and he gets to spend an interesting hour or so with someone who gives him the benefit of their insights and opinions.

The same is true, on a much larger and longer scale, for test managers. As we’ve stressed throughout this book, testing is a matter of providing useful services to stakeholders. If those stakeholders have a good relationship with you and the other test managers in your test group, information will flow more smoothly in both directions. The job of the test group will become easier because it has better access to information it needs. The test group will also become more valuable because the information the group produces will flow more smoothly to the recipients of that information. It’s just human nature: We listen to and value the communications we receive from people we are comfortable with, and we are happy to reciprocate that flow of information.

It’s not that you must be a personal friend to every stakeholder with whom you work, but a good professional relationship with those stakeholders is a major factor in the success of a test manager. How well you and the other managers in the test group initiate, cultivate, and sustain these relationships will strongly influence the flow of information, as well as the support, you obtain from your colleagues.

Rex notes that this anecdote (see callout on the left) does not represent an isolated incident but rather a truth that has become plain to him throughout his career in testing. The successful test manager, perhaps more than any other managers in the software business, must cultivate strong relationships with stakeholders, continuously reinforce those relationships with mutual benefits, and maintain the relationships through good times and bad. In the next few subsections, let’s look more closely at how.

4.5.1 Random Relationships

While going to work and going to college are very different experiences, a workplace is not terribly different than schools or universities in one interesting way: You will be placed in a communal setting with people, some of whom you probably didn’t know before, in an almost random fashion. In either situation, you get to choose how you approach the relationships you build and maintain, whether in college or at work.

Here’s where there’s an important difference: In college, you don’t have to build a relationship with anyone and you can still succeed. You could choose to go to your classes, spend all your free time studying, skip the parties and the games and the social events, talk only to your professors and teaching assistants about the material you are studying, and be a phenomenal success (at least academically). While this could be a sad but feasible path toward straight A’s in college, such an a social approach to any form of management, but especially test management, will simply not work. You don’t have to drink beers and go to football games with your coworkers to get ahead, but you need to make these random relationships mutually beneficial. You won’t succeed at that task if you approach it like some heartless cyborg or, worse yet, like a selfish manipulator.⁶

Why are these relationships so essential? Consider the following example. You are working as a test manager on a project to develop a new piece of software. That software will be built by a development team, so you need to communicate regularly and honestly with the development manager about the project. You’ll want to ask about the status of the development and where the team stands in terms of the lifecycle. You’ll want to discuss what level of quality the team can deliver to you, what level of quality the organization needs to deliver to customers, and how testing can help meet that goal. The development manager will want to know what kind of metrics and reports your team can provide. You’ll each want to know about deliverables you’ll exchange, along with details like how and when those deliverables will arrive, what should be done if there are problems with the deliverables, and so forth. You and the manager have a lot to talk about, and those communications will be much more pleasant and effective if your relationships are good.

As the anecdote about the vendor’s slimy CEO illustrated, this need for relationships is not just a within-the-organization thing. You need a good relationship with your vendors too. Of course, this is complicated by the fact that contracts proscribe limits on communication and also the fact that some testing service providers and other vendors are very good at putting glib and charming people in client-facing positions. It may take a while to learn how to juggle this balance, but it is possible to be professional, to fulfill your obligations to your employer and client, and at the same time have positive and enjoyable relationships with vendor representatives. And, if you are on the vendor side of the relationship, be a professional: Treat the client personnel you interact with exactly the same way you’d like to be treated, and don’t try to exploit the relationship.

Relationships—good or bad—are built through interaction. A major element of interaction in professional settings is communication. Some of the communications that will occur are formal communications, such as test results reporting. However, you should not rely on these formal communications alone to build strong relationships because people might see these as the test group simply doing their job. The best relationship builders are events that show you as a person willing to go beyond your obligations, to make a special effort to connect with your colleagues.

What are good ways to connect with colleagues as people? We have worked with clients and colleagues around the world and can attest to the fact that the right answer to this question differs culturally, organizationally, and individually. Here are some ideas.

Share a meal: This one sounds simple, but in fact, it is a very human way to make a connection. Many animal, even herbivorous animals, will tend to fight over food. Humans are, if not unique, certainly special in their ability to make meals a social activity where friendships are formed rather than a contest. You can choose to make the conversation social or about business, as the situation requires, but we have found that, in most cultures, setting aside business to talk socially with your colleagues will make a deeper connection. After all, you can talk about business in a meeting. However, if your colleague decides—at the beginning of the meal or at some later point—to talk about business, it’s a good idea to follow that lead. In some cultures, a shared meal, followed by business talk, is a major vehicle for problem-solving.

Share a drink: This is a somewhat weaker, but easier, variant of the previous technique. As before, our experience is that a social conversation is a stronger relationship builder than talking about business, so follow your colleague’s lead on topics but don’t jump directly to business unless you’ve agreed to do so. What you drink depends on the culture and the company, of course. In the United States, coffee, tea, or other nonalcoholic drinks will often be the only option during normal working hours, but it is not uncommon to have beer or wine with colleagues in Europe and Asia in the middle of a workday, usually with lunch. Drinking alcoholic beverages after work is an established tradition in some cultures, and in some cultures it’s almost essential to relationship building, while other cultures and individuals frown on drinking alcohol for religious or personal reasons. Remember that your objective in this situation is to build a relationship: If your personal habits include knocking back a couple of whiskeys at the end of the work day, when out for a social hour with a colleague who doesn’t drink, be careful about engaging in behavior that will actually damage your colleague’s opinion of you.

Inviting stakeholders to testing meetings: One of the risks associated with independent test teams is the possibility of isolation and even alienation. As we stress throughout this book, most test groups should see themselves as providing valuable testing services to their stakeholders, which goes a long way to managing this risk. However, it can also help to make sure that stakeholders feel welcome to contribute and participate in the way testing is done. For example, if you are introducing a new technique such as risk-based testing, why not invite stakeholders to a brief discussion on how the technique works? Obviously, you need to consider whether the invitee will find the topic interesting; issuing invitations to every testing meeting to every testing stakeholder will send a message of disrespect for their time, not relationship-building.

Celebrate the successes: Project and program teams often have events such as post-project parties, retrospectives, dinners, and other similar events on major milestones. You and the test group should attend. These can be good opportunities to set aside any negative baggage that accumulated during a tough project, provided you and your group are positive contributors in these events.

There are many other ways to approach relationship-building, but these four—breaking bread together, relaxing together, being open with each other, and celebrating together—are keystones of human society that go back to the dawn of human history. Relationship-building is a matter of social networking, and that goes beyond Facebook friending and Twitter following. Indeed, trying to establish such “social networks” without first doing the groundwork of building a real human relationship, via these types of techniques, will often come across as insincere and exploitative.

Relationship-building is an ongoing activity, not a one-time activity. Those of you who are extroverts may cheer to hear that, while introverts may groan. It is indeed the special challenge of the introverted manager to learn how to enjoy this obligation, but it will pay benefits to you, not only at work but also in your personal life. You should maintain and grow your relationships with stakeholders as the organization changes, as people change positions, and as roles and missions change. Having strong relationships is essential to being a successful test manager. Without such relationships, your team will be less respected than it deserves to be (based on its contributions) and less productive than it could be (based on its intrinsic capabilities).

It’s an interesting irony of our business that relationships are so important, isn’t it? After all, computers can communicate with each other on an entirely factual basis, and—other than in Star Wars movies—there is no need for politeness in the “protocol ‘droid.” In software, one program that refused to interact with another program based on a bad relationship—say because the other program hogged the CPU or memory—would be considered buggy. However, as long as software engineering remains a human endeavor, “courtesy . . . the lubricant of human relations,” and indeed relationships themselves, remain essential to its working.⁷

4.5.2 Who You (Should) Know

So you need to have good relationships to succeed as a test manager. But with whom? If you are in any organization larger than a small start-up, you can’t know everyone personally. From a purely Machiavellian perspective, even if you could, you probably don’t need to know everyone. Who should you know, and how well? The answer to this question will depend on the organization, but here are some thoughts.

Development and other peer managers: These people are the managers that you interact with on a regular, perhaps daily, basis. Development managers, of course, deliver the software you need for testing (possibly indirectly, through a release manager), and your group will be passing an evaluation of that software to the development managers, their developers, and other project stakeholders. Other peer managers on the project may also work closely with you and your group. These managers are probably also the people who are most influential in terms of the organization’s perception of you and your group’s effectiveness, which, as we’ve remarked elsewhere in this book, is a reality that you need to manage. It would make sense to get to know these people well and build strong relationships with them.

Technical documentation: It’s easy to forget about these people, but in organizations where technical documentation teams exist, they are key stakeholders for testing, with test-related interests similar to development managers.

Program and project managers: These people are often peer managers included in the first element in this list. However, if they are not, you should treat them as if they were, within the constraints mentioned earlier about upper management if applicable in your organization. Rex recently did an assessment where every program manager and project manager he interviewed had words of praise for one particular test manager in the test organization. After delivering the assessment report, he went to lunch at a Japanese restaurant on the way back to his hotel. Lo and behold, he meet her there—having lunch with a key project manager!

Sales and marketing: If you work in an organization that builds and sells software or software-based services, the people are very much interested in—and affected by—the quality of the software. They will have strong opinions about what you should test and how well your group is doing your job. (If your organization builds software for internal use, business analysts have a similar interest in testing.) As with peer managers and project managers, a strong relationship with key people and managers within these groups is essential.

Technical support: These people and the sales and marketing folks have a very similar interest in testing. They want to be sure that software is well tested, and they want to know the quality of the software before it goes to customers or end users. Rex has often found a strong relationship with technical support managers to be essential and very useful. He once managed to get the test scope expanded, and additional resources to do the testing, based on strong lobbying by a technical support manager.

Upper management: These are the people who approve your budgets, who write your performance evaluations, and who ultimately measure the success of your team. You definitely need a good relationship with upper management, but you must approach the task sensitively. An overly forward attempt to win over your superiors through meals or drinks will almost certainly come across badly with your peers, undoing any relationship-building you’ve done there, and will probably not work with the manager either. However, by celebrating successes with upper management, and assiduously ensuring positive and friendly professional interactions with all upper managers with whom you interact, you can establish a reputation as a valuable member of the team.

Vendors: In some cases, you may be testing software or systems provided by vendors. If so, strong relationships with vendor managers—especially test managers, development managers, and program or project managers—are essential. As illustrated in the opening anecdote of this section, a strong relationship with vendor counterparts can provide you with invaluable insights.

Key individual contributors: These are senior individual contributors, such as developers, system designers, business analysts, support staff, and even sales or marketing staff, who work regularly with you or your testers. Many IT organizations are egalitarian meritocracies, and the organizational structure and job titles do not completely reflect how power is distributed. People who have earned respect through doughty hard work and intellectual prowess will have a lot of influence on opinions, including opinions about you and your test team. These are also people to have good relationships with. Inviting them to participate in testing meetings will probably be a better approach than the “meals or drinks” approach because it fits better with their perceived thought leadership in the organization.

Key stakeholder representatives and other stakeholders: As discussed in Chapter 1, there are potentially many other stakeholders for a testing group. You should consider other people (whether managers or key individual contributors) who contribute to or influence testing or your group’s test results and quality assessments. If you went through the exercise discussed in Chapter 1, where stakeholders were identified, you should know who the test representatives or points of contact are for you. You should cultivate a relationship with these people as well, based on their importance to the actual and perceived effectiveness of the testing group.

Contrary to what you might expect, this job of relationship-building becomes more challenging as you move up the organizational structure. This is due to the fact that the individual test manager, working on a single project, will know and can build relationships with all of their coworkers on that project. However, as you move up into director or vice president positions, it will become difficult to know all the members of all the various project teams, and it would indeed probably come across as phony if you tried to do so. So from the broad menu given earlier, you must select the right people to build relationships with. Certainly that should include the peer managers and key individual contributors from that list. In addition, if there are people who have information that your test group needs or who control access to resources you test group needs, you need relationships with them as well.

In addition, you should not confine your thinking about whom you should know to those within your organization. Think about external relationships too. While the approach will often be different, and certainly influenced by various niceties of the business relationship, you should be just as careful to cultivate people outside your organization who are all the same important. If you ever catch yourself thinking, “Who cares what that person thinks about me; he’s just a vendor employee,” stop yourself immediately. Dehumanizing people not only is a moral issue, but also creates barriers to communication with people you need information from. Furthermore, it can undermine your other relationship-building efforts when people see how you behave toward others when you think nothing is at stake. As American humorist Dave Barry put it, “A person who is nice to you but rude to the waiter is not a nice person.”

You will probably find it easier to build relationships with some people than with others. Your social antennae may well push you toward going for the easy wins in terms of relationship-building and ignoring the others. Don’t let this impulse convince you to skip the harder cases, for two reasons.

First, there is a management cliché that says, “ ‘I like you’ often means ‘I’m like you.’ ” In other words, people often tend to prefer friendships with people who are similar to them in outlooks, personality type, and so forth. (Unfortunately, that can extend to gender and race, which has broad implications for test managers, including in hiring.) Since you manage a test group, which is a recipient and a source of information, you should be sure to avoid any sort of confirmation bias that could occur by building relationships only with those predisposed to agree.

Second, it’s no great feat to make a friend of someone who is already inclined to be your friend. Making a friend of a skeptic or an enemy, now that’s an accomplishment. Converting a skeptic or enemy not only makes your life easier by changing a negative to a positive in your political balance sheet, it also makes the organization as a whole more effective and efficient. In Rex’s years of doing assessments for clients, he has seen few things more generally corrosive to organizational success than negative personal relationships between key participants. He has become so convinced of this truism—mutual respect enables organizational success, and vice versa—that this dynamic is one of the first things he examines when he does assessments, and keep in mind that Rex has a degree in engineering, not psychology.

In case it’s not obvious by this point, your objective in this relationship-building and -maintaining exercise is not about assembling a coterie of drinking buddies or personal friends. It’s not about manipulating people. You want to make sure you and your test group are visible contributors to the organization. As a senior test manager, you must represent the test group in a way that wins respect and shows you as a strong, able, and emotionally intelligent leader.

We want to close this subsection by making an important point. Good relationships are important. Good relationships will make it easier for you and your team to do your job, and to do it better. Good relationships will also, frankly, buy you some political cover if there’s a big witch hunt for a scapegoat (pardon the mixed metaphor) after some particular disastrous quality issue during testing or in production. However, good relationships are no substitute for competence. If your intention is to cloak your inabilities with sycophancy and false friendships, that is unlikely to work in the long term. As Abraham Lincoln put it, “You can fool some of the people all of the time, and all of the people some of the time, but you can’t fool all of the people all of the time.”

4.5.3 A Real Social Network

As a manager of a test group, you cannot—and should not—be the only networker. Key people in your group should also form relationships across the organization, and you should be aware of those relationships. That way, members of the test team can act as your emissary. Effectively, you are delegating some of your relationship building to your team. This not only frees you up for other duties, it also teaches key people within your team the importance of this key management task, thus aiding them on their career path. In large organizations, or organizations doing distributed work, this “one degree of separation” approach to relationship-building can be not just a good idea but also necessary.

4.6Advocating Quality Activities Across the Organization

One problem we see in a few organizations is the “throw it over the fence and let the test group find all the bugs” attitude toward quality. In other words, developers and other project participants assume that one level of testing, right at the end of development, is sufficient to achieve quality. Of course, testing and quality tasks should be pervasive, spanning different groups and the entire software lifecycle. Part of managing across the organization is working with testing and quality stakeholders to make sure they understand the role other groups play in testing and quality when they define, create, and support software.

The specific roles these other groups play will vary from one organization to another. The lifecycle being followed matters too. So does the way in which the organization uses outside vendors and service providers, if any. However, here are some common examples to consider.

Reviews: Industry studies show that a substantial percentage of defects are introduced in precursor work products such as requirements specification, user stories, software design, database design, architecture specifications, acceptance criteria, and user documentation.⁸ Defects can also be introduced in test documentation. As explained in the Foundation and Advanced syllabi, and as borne out by numerous time-and-motion studies, the effort required to remove these defects is minimized when these defects are found and removed as close as possible to the moment of their introduction. Therefore, project stakeholders should participate, as appropriate, in reviews of these work products before they are used as the basis for subsequent project work.

Developer testing: There aren’t a lot of categorical statements that can be made about software testing and quality, given the highly contextual nature of software development, but there are few practitioners who would contest the assertion that developers should unit-test their own code before they call their code ready for a wider audience. Rex’s personal opinion is that this testing should achieve at least statement and branch coverage as well as covering whatever unit-specific risks the developer is aware of. In a number of organizations, developers are also responsible for component integration test. Ideally, testers (with a sufficiently technical background) would also participate in this level of testing. He has also seen a number of situations in which developers were able to contribute significantly to automated testing frameworks and even automated tests that were useful for system test and system integration test.

Validation, user acceptance testing, and operational acceptance testing: No matter how good the reviews are, no matter how good the developer testing, and no matter how good the testing your group does, some risk remains that the software built is not quite what the user needs. In addition, even if the software is what the user needs, the level of user confidence might not be what it should be at the end of the development process. To address these issues, business analysis, marketing teams, or others who understand the needs of the customer might need to do some validation testing during the system test or system integration test levels. Users, customers, or their representatives should be involved in user acceptance testing for IT software. Operators, such as system administrators, network administrators, and database administrators, should be involved in operational acceptance testing. When you’re building software that will be sold to a wide audience, such as mass-market and enterprise software, alpha testing and beta testing are ways to carry out this acceptance testing.

For optimal effectiveness and efficiency, the test manager should be—at the very least—aware of these other activities and what they will cover. This awareness allows the test manager to avoid retesting areas already adequately tested by others as well as to ensure that areas not tested by others are tested by their group. The best practice—which is far from the common practice—is for the test manager to work with the managers of other project participants to plan which testing tasks are best done at which points in the lifecycle. This determination allows the proper people to carry out these tasks, thus maximizing effectiveness, as well as to remove defects at the optimal point, thus maximizing efficiency.

In addition, the test manager should be provided with the results of these other testing and quality activities as they proceed. A comprehensive understanding of these activities, including the effectiveness of the testing done by the test group itself, will allow the test manager to understand and present stakeholders with an entire testing and quality picture. Suppose you are working with a client to develop a test strategy that includes a comprehensive test coverage plan and test results reporting process. While it’s not trivial to implement, the business case for doing so is immense. In engagements where we’ve examined the costs associated with suboptimal defect removal, we’ve found as much as 25 percent of the software development budget was wasted due to avoidable costs.

When this understanding reveals breakdowns in the total process for testing and quality, the test manager should be ready to analyze the causes, explain those causes, and present ways to fix the problem. For example, defects discovered late in the process, such as during user acceptance testing, could indicate inadequate reviews, gaps in the system test or system integration test, or problems with test environments of data. Human psychology and brain structure leads us to jump to conclusions when presented with problems, but the wise test manager knows that proper root cause analysis should be done carefully and with data. To paraphrase Jerry Weinberg’s Rule of Three, if you haven’t thought of at least three different ways in which a process breakdown could have occurred, you probably haven’t thought enough.⁹

The analysis of other groups’ testing work can be complicated by differences in record keeping. Independent test groups, composed of professional testers, tend to have detailed documentation and rigorous processes for managing test work products and results such as tests, test basis traceability, and, especially, defects. Other groups involved in testing are typically much less detailed. In the ideal case, the management of all testing work products and results would occur in the same test management tools, but that is seldom the case. At the very least, if a consolidated view of testing is needed, the test manager must work with other groups to define what information must be captured by each group and how that information can be collated.

It is by no means unusual for other groups to commit to perform testing tasks but to perform them poorly or not at all. We regularly hear testers and developers talk about unit testing that is not performed, or is performed by some developers but not by all developers, or that is performed by developers who don’t understand the fundamental concepts of unit testing. For example, many developers do not have a good understanding of code coverage.¹⁰

It is typically not the test manager’s job to enforce adherence to software engineering best practices. In addition, it’s usually not the case that these other groups have omitted or short-changed these testing tasks because they are pathological liars who enjoy telling people they will do things and then shamelessly blowing off those commitments. Tight schedules, scope creep, and resource issues often mean that project pressures limit what these groups can do. Further, if testing tasks are seen as low priority, or perhaps not so critical because after all a test group exists, they’ll be the first tasks dropped under pressure.

In some cases, especially when the problem is not omission of the testing tasks but rather incompleteness, the underlying problem is a lack of know-how. As mentioned previously, many developers don’t know the basics of code coverage, so their unit testing will not tend to achieve even 100 percent statement coverage. Sometimes the problem is one of motivation, especially when people are being asked to do testing tasks they have not done previously. In fact, we have seen cases of outright demotivation by managers, where perverse incentives result in testing tasks not being done or test results not being recorded.

Whatever the cause, it is important that the test manager knows what has and has not actually been tested. This assessment should be done through proper verification of the work. For example, it is not enough simply to ask the development manager, “Did your programmers unit-test their code?” The answer to that question will almost always be yes, but that answer is not always connected to reality.

Assuming you find that certain tests have not occurred, what can you do? In the long term, the answer is to work with your fellow managers, and with executives, to solve the underlying cause. If management knows to what extent constraints, prioritization, lack of knowledge, and lack of motivation are resulting in the testing gaps, a plan can be formulated to resolve those issues and enable better testing across all the participants. How this plan is created, and who does what to implement the plan, will vary across organizations. Some of our clients have found that Agile lifecycles in particular enable test managers to effectively advocate pervasive quality activities.

In the immediate term, if you discover that some planned set of test activities did not occur, you need to make the best of a bad situation. If you discover the gap early, prior to the start of your own group’s test execution role, you have more options. You can, perhaps, work with the team that is experiencing difficulty completing their test tasks in an attempt to salvage the work. While this might have the effect of diverting your resources and reducing your group’s effectiveness and efficiency, it could be that, for the overall effectiveness and efficiency of the organization, this is the best course of action. If you discover the gap after your test execution has started, you can try to obtain more resources and/or time for testing. As you make the choice about how to respond, remember that the proper balance of quality, schedule, budget, and feature choices must be optimized on an organization-wide basis.

4.7Integrating Tools Across the Organization

In the previous section, we mentioned the advisability of sharing test management tools across groups involved in testing as a way to enable better coordination of testing and reporting of test results. This is but one example of how tools can be—and frequently are—shared across organizations. While sharing tools has obvious appeal, great care and planning is required for this sharing to work.

For example, we’ve seen a number of situations in which organizations are using the same tool to track defects, development tasks, and requirements. This tool is often one best suited to tracking tasks or user support incidents, thus making the out-of-box defect and requirement tracking abilities limited. This is not to say that the tool cannot be made to work, but a number of modifications will be necessary. For one thing, it’s necessary to distinguish between the three quite-different types of data being managed in the tool. It’s also necessary to gather different types of classification information depending on the type of data in question. This will allow analysis and reporting of defects to occur in a meaningful fashion.

This type of well-structured multiuse tool implementation cannot happen by accident, and it is much harder to put the structure in place after habits of usage have been established and, often, significant repositories of dirty and confusing data have been accumulated. When tools are bought (or downloaded for free) and then turned loose in the organization, their usage can grow like a weed, with no documentation. It’s difficult to maintain, support, and update such tools. Furthermore, if there is a need to convert data to or from such a tool or to migrate tools, the opacity surrounding the tool makes such conversions and migrations inefficient and error prone. Instead, the best practice is to acquire, implement, and use these tools according to a predefined (though often evolving) plan. Let’s examine what we need to plan for across the tool lifecycle.

4.7.1 Purchasing, Selecting, and Acquiring the Multiuse Tool

Think back to the last time you went to lunch or dinner with a group of friends or your family. In this situation, it would be unusual for one person to order for everyone, and if they did, some of the diners would probably not be satisfied with their meal. Instead, people choose for themselves. In some workplace cases, there might be constraints, such as limits on what could be purchased (e.g., alcohol may be precluded in a company lunch) or what the total amount for the meal could be. In a sense, the diners are a selection committee.

In the case of a multiuse tool, you also need a selection committee. Just as every diner must participate in the selection of the meal, every stakeholder in the purchase, use, and maintenance of the tool must be involved in the committee. However, there are many more requirements, constraints, and risks to consider. At least in this selection committee, unlike on a family trip to a restaurant, you probably won’t have to tell your five-year-old to stop putting carrots up their nose!

Part of the job of the selection committee is identifying the requirements, constraints, and risks associated with the tool. Each of the stakeholders on the committee has insight into some of these items, but people outside the committee might need to have input as well. The well-led committee is careful to understand these requirements, constraints, and risks before taking any further steps.

Some of these requirements, constraints, and risks involve integration, in terms of both integration with other tools and integration with the processes. Each point of integration must be defined. All of the data to be captured and shared must be clearly identified and defined, as must the ways in which this data is to be stored, received, and transferred. If there are any customizations required, especially custom programming or database schemas, they must also be identified and defined, including the responsibility for implementing and maintaining the customizations.

Of course, the multiuse tool must be used by people in various roles. What different user groups exist, and what will they use the tool to do? What administrator roles exist, and what must they do to enable secure, reliable, recoverable use of the tool? How will the users be managed? Are there license issues associated with the users? The selection committee must find answers to these questions.

Another critical area of consideration is ownership and responsibility. For example, if the tool is a commercial one, it must be paid for. That money must come from someone’s budget, and the purchase must be approved. Even open source tools have approval issues, because the agreements under which they are distributed (such as the GNU General Public License and the Creative Commons license) often impose responsibilities on the organization. As the committee plans for the implementation project and the long-term maintenance of the tool, clear responsibilities must be assigned. Ideally, as with any project, a project manager is assigned for the implementation work.

4.7.2 Updating, Maintaining, and Supporting the Multiuse Tool

Once the implementation work is underway, there can be a considerable amount of programming and testing involved, depending on the degree of customization and integration required. Many of the best testing tools available these days are multiuse tools or can be integrated with other tools to create a multiuse tool. As such, planning and performing this implementation work often requires significant tool project management and technical expertise that might or might not be present in the organization. Rex and his associates have worked on a number of engagements with clients where RBCS associates provide this expertise.

Software engineering tools, including testing tools, are all forms of software, which makes them changeable. As every test manager knows, this malleability of software is not without risk and must be managed. A clear change management process for multiuse tools is critical because otherwise, each stakeholder group’s needs and desires will often create centrifugal forces that degrade the overall usefulness of the tool. At the same time, the change management process must not make change too difficult because in that case changing needs cannot be accommodated.

For some tools, it is possible to make localized customizations that affect only certain user groups. Theoretically, these localized changes do not have the same centrifugal tendencies that overall changes do, but that’s not always the case. If, for example, developers decide to modify the classification fields for tasks to include a priority field, that can be confusing if the defined levels of priority differ from those used for defects. For all changes, it’s important that impact analysis be done to identify whose use cases will be affected and how.

As mentioned earlier, multiuse tools have a disparate base of users, each with their own particular use cases and abilities. A common mistake is to fail to provide support for these users. Simply providing a tool, even if the tool is perfect, is not enough. People must be trained and supported in the use of the tool. Since, in many cases, these multiuse tools support critical team activities, there must be well-defined service-level agreements (SLAs) in place. For example, if the defect tracking tool were inaccessible or just highly unreliable during test execution for three days, that would likely disrupt the test group considerably. On one project where developers selected the defect tracking tool without input from the test group, the test group suffered from unreliable access to that tool for months, resulting in persistent inefficiencies.

During or after the implementation, it might be necessary to migrate data into the tool. This could include defect reports from a previous tool, tests documented in spreadsheets, or even unstructured data. This must be done with utmost care. It is often the case that production data is riddled with errors, so you should plan to deal with records that contain such data quality issues, especially those having to do with referential integrity. For example, if each test must be related to one or more test basis elements (e.g., a requirements specification element or quality risk item), the process of importing tests from spreadsheets must include some way of establishing this relationship.

For many tools, large quantities of data will be managed over a long period of time. Data related to defects, tests, test results, and test coverage form the raw material from which project, product, and process metrics are derived (see Chapters 6 and 8). Therefore, an additional data quality concern is that of recoverability. In the event of a disaster, ranging from a disk failure all the way up to the loss of a data center, it must be possible to restore the data. This entails clear requirements and responsibilities for backing up, restoring testing, and actually restoring data (if the need ever arises).

4.7.3 Converting and Retiring Tools

When Rex started as a tester in an independent test lab, the state of the art in defect tracking and test management in his organization was a Lotus Notes database. He and his colleagues captured all their test results in that repository and used it to produce reports for clients. Later, they added a Lotus 1-2-3 spreadsheet as a way of summarizing test status in a single consolidated view. Rex thought he had a pretty good set of tools and processes for test management, and it did serve his organization’s needs.

The best practice now, of course, is miles beyond such homespun tools. Of course, as sophisticated as test tools are today, there’s every reason to think that the evolution of tools will continue. Business needs evolve. Some tools vendors thrive while some vendors fail. Sometimes external factors prevail, such as when a client insists that a testing service provider use their tools on their projects. So don’t get too attached to any given tool because conversions will happen.

These conversions are never as simple as flipping a switch. Consider whether some data must be migrated from the incumbent tool to the new tool and, if so, how that will be done. Users staying on the incumbent tool (if any) will need to retain access to it without any interruption. The users migrating to the new tool should see minimal disruption of their service and consistent data, before and after the migration. It’s highly desirable that the conversion be to a tool that is at least as usable, reliable, and functional as the incumbent tool.

If all users are to be moved off the incumbent tool, then the tool may well be retired. Retirement of a tool should not occur unless a new tool is fully available to all the users, unless there were no users of the retired tool. The key is that any decision to convert or retire tools should allow (ideally seamless) continuous availability of the relevant business function for all users.

In some cases, a tool or some portion of it must be retained for ongoing access to the original data, though it’s typically preferable to migrate the data to the new tool and access it there. However, it might be desirable to retain a utility that can secure and access archived data without putting it into the live dataset of the replacement tool. For example, if the amount of archived data is very large, it might create performance issues to load it.

4.7.4 The Manager Role with Multiuse Tools

What is the test manager’s role with multiuse tools? It depends, often more on accidents of organizational history than any deliberate or rational choice. As a test manager, you should be prepared to act as the owner of the tool or as one of the stakeholders. Either way, you and your team need reliable access to the tool, and accurate test-related data in the tool, while at the same time so do the other stakeholders.

If you are the owner of the tool, your job includes understanding and accommodating the needs of the other stakeholders. The implementation of the tool should accommodate all stakeholders and avoid forcing trade-offs where improving the utility of the tool for one group degrades the utility for another group. This imperative to support diverse needs and users in multiuse tools creates complicated issues beyond those associated with single-use tools used only by testers.

Given the complications, there is a considerable time and effort associated with owning these tools. As was explained in the preceding sections, there are many issues that must be addressed with careful forethought in planning for the implementation, use, and support of such tools. Because the plan must span multiple stakeholder groups, the owner of multiuse tools must be adept in organization politics and have a diplomatic approach to the tools’ stakeholders. The owner must be ready to negotiate and compromise to obtain support. However, these compromises can have negative side effects for some of the stakeholders, so creativity is often required to avoid that.

In some organizations, the potential conflict of interest associated with a stakeholder group’s ownership of multiuse tools is avoided by forming a separate tools group. This group is truly a service organization and must act like one. It should define usage guidelines that reliably enable the key business functions supported by the tool while preserving security, recoverability, and data quality.

4.8Handling Ethical Issues

In an earlier section, “Creating and Building Relationships,” you read about a fellow so ethically challenged that one of his colleagues felt the need for a shower after meeting with him. Now, clearly this fellow lay toward the extreme end of the ethical spectrum, but not one of us can claim to be perfectly ethical. In some cases, we lapse ethically due to ignorance or momentary weakness. In other cases, a herd mentality exists, where we see unethical behavior around us go unpunished—or worse yet get rewarded—and so we adopt an “everyone is doing it” attitude.

As a manager, you are responsible not only for your own ethics, but also for those of your team. If people on your team are behaving badly, you must not tolerate it. Further, you must set an example of ethical behavior for your team. Challenges to ethical behavior can arise in many different contexts within a project and an organization, so you should always be on your guard.

4.8.1 Managing the Team’s Ethics

The ISTQB code of ethics is defined in the Foundation syllabus. For ease of reference, we’ve included it here.

Public: Certified software testers shall act consistently with the public interest.

Client and employer: Certified software testers shall act in a manner that is in the best interests of their client and employer, consistent with the public interest.

Product: Certified software testers shall ensure that the deliverables they provide (on the products and systems they test) meet the highest professional standards possible.

Judgment: Certified software testers shall maintain integrity and independence in their professional judgment.

Management: Certified software test managers and leaders shall subscribe to and promote an ethical approach to the management of software testing.

Profession: Certified software testers shall advance the integrity and reputation of the profession consistent with the public interest.

Colleagues: Certified software testers shall be fair to and supportive of their colleagues, and promote cooperation with software developers.

Self: Certified software testers shall participate in lifelong learning regarding the practice of their profession and shall promote an ethical approach to the practice of the profession.

These ethics apply to any interactions made by you or the test group, including people inside and outside the organization. As mentioned earlier, you need to hold your group to those ethics, but remember that this must be done fairly. People should be trained in the application of the code of ethics. Make sure the ethics are clearly stated to the team and that each person understands that they are individually accountable to them. Tell them, and show them, that the herd mentality will never be tolerated as an excuse in your group, regardless of what others are doing.

Reinforcing ethics is an ongoing activity. As you carry out annual performance evaluations, be sure to note and reward ethical behavior and conversely to censure unethical behavior. When you find a situation that has interesting ethical implications, use that as a case study for your group, examining and discussing acceptable and unacceptable courses of action.

Ethics are important for all managers, but especially test managers. Because testers are often the bearers of bad news, conflicts can occur, sometimes unexpectedly. This requires testers to keep their professionalism and their cool, regardless of the behavior and comments from those around them. A single outburst, ill-considered email, or ethical lapse in the wrong situation can undo much of the relationship-building that has been a core concern of this chapter.

4.8.2 Interacting with Test Stakeholders and Reporting Test Results

In your interactions with the various test stakeholders, you and your group must behave professionally, keeping personal interests and emotions contained. Tell the facts as well as you know them, accurately and objectively, while also communicating effectively. Consider the code of ethics, particularly those having to do with client and employer, product, judgment, management, and colleagues. People tend to trust and respect others when they respect not only the ends they achieve but also the means by which they achieve those ends.

As a test manager, you’ll typically produce numerous reports, metrics, charts, and presentations for testing stakeholders. Remember that the ethics statements relating to product mean that the data and the analysis you’ve done on the data are accurate. That also means that you should present test results clearly and correctly, because information is not successfully transmitted until it is successfully received. Further, the ethics statements relating to your colleagues mean that reports should not target anyone. A classic mistake test managers make in this regard is producing reports that show the number of defects introduced by each developer or name the developers who own the defect reports that have been languishing the longest in an assigned and unresolved state. Classifications and reporting can be done by any number of impersonal groupings, such as features, quality characteristics, or modules, and using these groupings, you can often achieve the same informational result without making someone feel targeted.

These ethical considerations do not apply only to management-level communications. Defect reports, status reports, email, hallway conversations, instant messages, and indeed any form of written or verbal communication must be ethical. This means that the information is conveyed objectively and fairly, without malice or hidden agendas. If some recipients might misunderstand the information conveyed, providing context and background is required.

Because huge quantities of information come out of test groups and from every level of the test group, consistent application and promulgation of ethical standards are critical. Everyone’s communications either support or degrade the credibility and perceived value of test team. We have seen the damage done when one intemperate email, one stupidly expressed defect report, or one angry instant message found its way into the wrong hands, and thence brought to the attention of management.

Containing emotions is often a difficult standard to achieve. Test status and test results can be controversial, and people sometimes react defensively. Testing can be squeezed by project schedules, and it can seem like unfair micromanagement when other managers suggest throwing tests (and thus quality goals) overboard to hit a deadline. It can be particularly galling when you, as the test manager, always thought the deadline was a completely unrealistic fiction to begin with.

It’s important to remember that, as a general rule, the more you let anger and frustration enter into your communication, the less effectively you’ll communicate. Your emotional state becomes the main information received, distracting people from the objective realities of your testing. Instead, while preserving the integrity of your message, try to understand how each stakeholder will relate and react to it. If it’s going to be unpopular, so be it, but be prepared to handle the negative response you’ll receive.

4.8.3 Test Management Ethics

As a test manager, you have ethical concerns that are specific to being a test manager, but you also have ethical concerns that are general to all managers. As a manager, you are entrusted with the care of those people in your group. You must behave professionally and ethically at all times. While time and space preclude a complete discussion of the implications, here are some highlights to keep in mind.

Personal issues: It is fairly common for managers to find that their direct reports share certain personal details of their life. For example, someone may tell you that they will be late for work or working from home due to a sick child or that they will need to take a leave of absence for a couple weeks to deal with a parent’s terminal illness. Depending on your personality type, your organization’s culture, and the people in your group, this can occur frequently or infrequently, but some level of personal communication will inevitably occur. Who else you share this information with, and how, is an ethical question. In the case of someone working from home due to a sick child, you might mention that to coworkers to explain their absence. However, in the case of the death of a close relative, you should ask the employee first before sharing that information. Ask yourself, “How would I want this information to be treated if I were in my employee’s place?” That question will usually guide you toward the right ethical behavior.

Individual performance issues: If someone in your group is not meeting the expectations for their work or is behaving in a way that is disruptive, you have an obligation to your organization and to your test group to work with the person to resolve the underlying problem. This must be done in a way that is both fair to the individual who is underperforming and expeditious in returning overall test group performance to expected levels. You should handle the discussion of the performance issues with the individual discreetly, frankly, and with an honest intention to do the best thing for the individual and for the organization. Of course, gossiping about the individual’s performance issues is inappropriate, but you might need to discuss the situation with other managers who have observed or been affected by the substandard performance. Often, an organization’s human resources department will have rules about how to deal with troublesome employees, and you should follow those rules closely.

Intragroup interpersonal issues: This is a special case, and an especially difficult case, of disruptive behavior in your group. It occurs when two or more individuals in your test group develop a negative or distracting relationship. People don’t have to be friends to work together, but they do need to cooperate and behave in a professional, productive way in the workplace. This category of issues can also include the office romance, if organizational rules preclude it or if the romance spills over into open displays of emotion at work. You need to work with the individuals involved to resolve the interpersonal issues that exist or at least to get the individuals to agree to keep the issues from affecting their work performance and behavior. As with the previous item, discuss the matter only with other managers or HR staff who have a legitimate need to know and to be involved.

Legal issues: The apex of all regrettable ethical issues is achieved when those issues rise to the level of civil or criminal legal action—or potential action. Examples include harassment, retaliatory behavior, or discrimination based on gender, sexuality, race, or other protected class. The way you handle this as a manager will prove a true test of your ethical character. (Obviously, if you are the harasser, you have already failed that test.) If someone in your group approaches you and complains of harassment by someone else in your group, that is a situation you must act on, but you must act on it properly, not impulsively. First, get the details (in private, of course), acknowledging the report and assuring the person that action will be taken. Next, contact your human resources department, your organization’s legal department (if any), and your manager. If the behavior being reported is of a criminal nature—e.g., brandishing a handgun in the course of an argument—then you might have an obligation to involve the police, but you should be cautious about escalating to the police unless you are absolutely sure it is necessary. Now, you must remember that people have a right to a presumption of innocence, and all sides of the story should be heard. As you work with your manager and the human resources department and proceed through the resolution of a difficult issue, remember to be fair toward all parties, extremely discreet in your discussions of the matter, and sober and serious in your demeanor about the situation. Rex once observed the head of an organization making jokes, in a public setting, about a series of disturbing and abusive emails that were broadcast within the organization to all of the members. Rex didn’t have much respect for this person going into that situation, and with this display of utterly unethical behavior, which was a true window into his character, Rex lost all respect for him that remained.

The complexity of handling these ethically charged issues is complicated by the other stresses of your workload. While it’s never a good time to deal with the kind of issues mentioned, there are typically a score of schedule, budget, quality, and technical issues vying for your attention when these challenges arise. Nevertheless, you must have the ability to multitask as a manager. The kinds of personnel issues mentioned here rarely resolve themselves on their own, and even if they might, it is highly unethical for a manager to ignore such problems. As much as it is human nature to avoid difficult or distasteful situations, it is your job to handle them, promptly, professionally, and fairly.

In some cases, these issues become further complicated by the fact that the problem involves another manager’s employees. For example, if there is a developer who is writing code that causes lots of problems in testing, you might become aware of that individual’s performance issues before their manager does. We have seen other examples, such as hostility or aggression toward testers by people in other groups. These situations are the moments where all the relationship building discussed in this chapter will come into play. You’ll need to approach the other manager, discreetly and in private, to discuss the problem. If you are dealing with a situation where someone in the test group is being mistreated by someone outside the test group, be ready to spend some of the relationship and political capital you have accumulated on resolving that situation.

Dealing with human and relationship issues is an inherent part of a manager’s job. There’s no formula for simple resolution of these issues. When in the midst of resolving such issues, remember that you have spent a lot of time building relationships and your reputation as an ethical manager. Incompetent or callous handling of delicate situations can do immense damage to those relationships and your reputation. Carefully assess the situation before you act, and think carefully before you speak.

4.9Sample Exam Questions

In the following section, you will find sample questions that cover the learning objectives for this chapter. All K5 and K6 learning objectives are covered with one or more essay questions, while each K2, K3, and K4 learning objective is covered with a single multiple-choice question. This mirrors the organization of the actual ISTQB exam. The number of the covered learning objective(s) is provided for each question, to aid in traceability. The learning objective number will not be provided on the actual exam.

The content of all of your responses to essay questions will be marked in terms of the accuracy, completeness, and relevance of the ideas expressed. The form of your answer will be evaluated in terms of clarity, organization, correct mechanics (spelling, punctuation, grammar, capitalization), and legibility.

Question 1

LO 5.2.1

You are the test manager of a small but highly effective testing group. In a discussion with the owner of the company, she tells you that the size of the programming team will double over the next year. She then asks you whether you can continue to operate with the current test team staffing level and still improve efficiency of testing.

Which of the following statements demonstrates how best to advocate the test team in this situation?

1. Use metrics to show the owner the likely reduced effectiveness of testing and thus increased cost of quality.
2. Tell the owner that this plan should be possible if sufficient investments are made in training the test team.
3. Be forthright in admitting the mistakes the test group has made, and explain how you intend to correct them.
4. Explain to the owner that preserving an industry-standard tester-to-developer ratio is essential to the organization’s success.

Question 2

LO 5.2.2 and LO 5.3.1

Refer to Scenario 1. Explain how you would propose to organize your testing group, and connect the proposed organizational approach to specific values that this approach would promote.

Question 3

Consider Scenario 1.

LO 5.6.1

Which of the following statements lists other groups you would expect to be involved in testing, in their order of involvement in the project?

1. Requirements engineers, developers, testing services group, executive management, selected customers.
2. System architects, requirements engineers, developers, testing services group, selected customers.
3. Developers, testing services group, selected customers.
4. Requirements engineers, system architects, developers, testing services group, selected customers.

Question 4

LO 5.2.3

Consider the following situations:

1. Project leadership asks for an analysis of why the number of bugs found exceeds the estimate.
2. People use cost-of-quality metrics to determine the efficiency of the test team.
3. The number of testers assigned to the test team is reduced after test execution starts.
4. Testing stakeholders give the test team input on what should be tested.
5. Testing stakeholders ask to participate in a quality risk analysis session during test design.
6. The development manager says testing is taking too long because too many bugs are being found.
7. Executive management expects to receive regular updates on defect detection effectiveness.
8. A project manager tells the test manager which testers should be assigned to a project.

Select three of the above situations where a test manager would need to defend the test team.

Question 5

LO 5.4.1

Consider the following list of testing work products:

1. Detailed defect reports
2. Test summary reports
3. Test process metrics
4. Requirements coverage report

Consider the following list of testing stakeholders:

1. Business analysts
2. Programmers
3. Project managers
4. Executive managers
5. Technical support staff
6. UAT manager

Select the answer that correctly matches stakeholders with the testing work products that need to be communicated to them.

1. A: II, V; B: III, IV, VI; C: IV; D: I, III, VI.
2. A: II; B: III, IV, VI; C: IV; D: I, III, V, VI.
3. A: II, V; B: I, II, VI; C: IV; D: I, III, VI.
4. A: II, V; B: III, IV; C: III, IV, VI; D: I, III, VI.

Question 6

LO 5.7.3 and LO 5.7.2

Consider Scenario 1.

Assume that senior managers have directed you to implement a defect tracking tool that will provide better visibility into defect introduction, detection, and removal in the software engineering process. This will involve having a single tool that will be used to track defects at all stages of the lifecycle, including support defects. An existing tool is used to track requirements (in the form of user stories), development tasks, and defects found during development, and this tool has a large repository of historical defect data. You have determined that this tool will not meet the needs of the organization.

Outline a plan for selecting, acquiring, and implementing this new approach to defect tracking.

Assume that senior managers have directed you to implement a defect tracking tool that will provide better visibility into defect introduction, detection, and removal in the software engineering process. This will involve having a single tool that will be used to track defects at all stages of the lifecycle, including support defects. An existing tool is used to track requirements (in the form of user stories), development tasks, and defects found during development, and this tool has a large repository of historical defect data. You have determined that this tool will not meet the needs of the organization.

Outline a plan for selecting, acquiring, and implementing this new approach to defect tracking.

Question 7

LO 5.8.1 and LO 5.5.2

Consider Scenario 1. Assume that you have now implemented the unified defect tracking system described in question 6. During test execution, one of your senior testers tells you that a programmer asked her not to report three critical, potentially safety-related defects that she had located. The programmer told her that his annual performance evaluation would be affected by the defect reports because development management had set maximum defect limits for each programmer after the new defect tracking system was put in place.

The question consists of two parts:

1. Describe the ethical issues associated with this situation.
2. Outline a resolution that results in minimal damage to existing relationships between testers, programmers, test managers, and development managers.

Question 8

LO 5.5.1

Assume that you have recently been hired as a test manager in a growing software-as-a-service vendor. This organization has not previously had a testing organization but rather has relied entirely on programmers testing their own code and limited beta testing managed by the technical support team.

Which of the following is a good first step to start creating relationships in this situation?

1. Ask the programmers to participate in a brown-bag lunch session with the new testers to explain to the testers how they unit-test their code.
2. Schedule a meeting with executive managers, and give a presentation for them on how much more effective independent test groups are at finding defects.
3. Have lunch with the development manager and technical support manager, letting them know you’re looking forward to coordinating testing efforts with them.
4. Have drinks with the technical support manager, and ask him if he can help you understand what the developers are currently missing in their unit testing.

Question 9

LO 5.7.1

Assume that a company’s test management tool loses some of the classification attributes (e.g., priority) associated with a requirement when importing new and updated requirements from the requirements management tool. Which of the following options is a multiuse tool issue that should have been considered during the purchasing, selection, and acquisition of the test management tool, to help avoid such a problem?

1. Integration of data types
2. Ownership of the tool
3. Defined change management process
4. Continuation of service on tool retirement

1For the source of this data, see Tom DeMarco and Tim Lister’s book, Peopleware: Productive Projects and Teams (3rd Edition).

2The original reference is J. M. Juran’s Quality Control Handbook, a magisterial (and weighty in many ways) book first published in 1951. Rex’s own books Managing the Testing Process Practical Tools and Techniques for Managing Hardware and Software Testing/ Edition 3, Advanced Software Testing - Vol. 2, 2nd Edition: Guide to the ISTQB Advanced Certification as an Advanced Test Manager, and Critical Testing Processes: Plan, Prepare, Perform, Perfect include discussions of applying the technique to software and systems development. You can find an online explanation of the technique on the RBCS website at www.rbcs-us.com/software-testing-resources/library/basic-library in the article called “Testing ROI.” You can find a spreadsheet that will help you calculate testing ROI using that technique at www.rbcs-us.com/blog/2010/09/09/free-tool-for-calculating-software-testing-roi.

3Actually, DeMarco used a more frank and pungent US English word for lies and exaggeration, which we’ll avoid to retain a general audience rating for this book. This memorable quote on management, and other good ones, can be found in his book The Deadline.

4For two interesting perspectives on projects gone awry and why, see Ed Yourdon’s book Death March or, for a quality-focused perspective, Capers Jones and Olivier Bonsignour’s book The Economics of Software Quality.

5You can see Boris Beizer’s book Software Testing Techniques for the first mention of this problem we’re aware of. Rex wrote about it first in the first edition of Managing the Testing Process: Practical Tools and Techniques for Managing Hardware and Software Testing, and that material remains, revised to some extent, in the third edition.

6One of our reviewers, Capers Jones, wrote, “I’ve seen projects canceled because high-level managers did not like each other and would not cooperate.”

7We had heard this aphorism that “courtesy is the lubricant of social relations” many times before, without knowing the source. Quoting it here required that we determine where it first appeared, and someone found it in The Management of Men, by Edward L. Munson. Of course, we would suggest that courtesy is important whether managing men, or women, or both.

8See, for example, any number of Capers Jones’s books, including Estimating Software Costs: Bringing Realism to Estimating and The Economics of Software Quality.

9This rule appears in Gerald Weinberg and Virginia Satir’s book The Secrets of Consulting: A Guide to Giving and Getting Advice Successfully.

10You or others who want to know more about code coverage, or should know more about code coverage, can listen to Rex’s webinar Advanced Software Testing: Code Coverage, available at www.rbcs-us.com/software-testing-resources/library/digital-library.