3Managing External Relationships

Keywords: None for this chapter

3.1Introduction

There was a time, ages ago in Internet time but within living memory of grizzled software veterans such as your humble narrators, when everyone on a software project worked in the same building. Rex’s first experience writing software—and also testing his own software, which was how most testing was done when he started in the software business—was with a team of people in West Los Angeles who all sat within 50 feet of each other on the same floor of the same office building, most of them working in product teams in open spaces together.

Of course, even then, in the early 1980s, plenty of projects involved third parties, outsourcing, and other forms of distributed work. Rex and his colleagues created a customized version of the company’s portfolio management and accounting system for a client on the East Coast, Goldman Sachs. Conference calls, letters, and on-site visits were used (as this was BE—before email) to communicate project status, change requests, and the like.

However, it’s certainly true now that a test manager is much more likely—indeed, probably certain—to be involved in projects where a third party is involved. In this chapter, we’ll focus on situations in which this third party is a vendor or another group in a different part of the organization. We’ll assume that this third party is creating, testing, or delivering some or all of the product. We’ll discuss the specifics of this third-party relationship—which can be complicated—in the next section.

As a test manager, you need to ensure that the vendors deliver the proper level of quality. Working with third-party vendors affects the schedule, the test approach, the documentation, the communication channels and styles, and more. Facilitating smooth delivery of a quality product in these situations is a special challenge to the test manager.¹

3.2Types of External Relationships

Let’s start by examining the various types of external relationships that can exist and affect your job as a test manager.

The first type of external relationship is where a third party delivers a completed software product or system. This is sometimes referred to as turnkey development. This can run the spectrum from the purchase of commercial off-the-shelf (COTS) software to custom development based on the customer’s specification. In addition, the system itself can run the spectrum from being used in a purely standalone fashion to being tightly integrated with other systems in the customer’s data center. Examples of this type of relationship include purchase of a custom-developed loan origination package or an enterprise virus-protection package.

The second type of external relationship is where a third party does only the software development, with the testing left to the customer or recipient. In our experience, even when the relationship is supposedly a turnkey one, developing organizations often perform inadequate testing. So, this type of relationship explicitly allows for the deliberate detection of defects by the recipient, with (one would hope) some mechanisms for defect repair, confirmation testing, and regression testing. An example of this type of relationship is an organization having an e-commerce site custom developed.

The third type of external relationship is where a third party develops a subsystem or module that the customer will integrate into an overall system. The difference between this type and the first type is that, in the first type, a complete system is delivered, where in this type, the delivered component must be integrated into a larger system. An example of this type of relationship is an organization purchasing a business-card scanning subsystem to include in its sales management application.

The fourth type of external relationship is where a third party will handle the testing of a product that has been developed internally. This typically would include the higher levels of testing and less frequently might include unit or component integration testing. An example of this type of relationship can occur when a start-up company—or just a few smart programmers—creates a software product but has no testing staff, environment, or capabilities, so it hires an outsource testing service provider to perform all the testing.

The fifth type of external relationship is where a third party and an internal test team each do some of the testing of internally developed software. As with the fourth type of relationship, the work done by the third party typically involves higher levels of testing. This might seem like a variant of the fourth type, but it tends to be more complex (at least when done well) due to the need for coordination between the groups doing the testing. We’ll discuss the issues associated with integrating test strategies across multiple organizations in a subsequent section.

On any given project, more than one of these types of external relationships might exist. In addition, external relationships might exist that contain elements of more than one type. The purpose of presenting these relationship types is not to provide a way for you to perform a clean, one-to-one classification of actual business relationships on a project but rather to help you think about these various types, and, in subsequent sections, how the properties of these relationship types affect testing.

The type of external relationships that exist will strongly influence the test work on the received product and the organization of that work. Obviously, each additional party involved in a project increases the importance of good communication, proper documentation, and appropriate means of coordination between the parties. We’ll discuss these issues further later in this chapter, but first let’s look at how contractual matters affect these external relationships.

3.3Contractual Issues

Regardless of the relationship type, in most circumstances we’ve seen, working with an external entity involved a contract. In the best case, you as the test manager will have input into the drafting of the contract, at least on those elements of the contract that involve testing. In this fortunate circumstance, you will be consulted on the implications for the testing process and be able to contribute to certain technical and methodological aspects that will affect testing and quality. In most cases, though, the legal department or senior management will handle the contract. In some cases, we have seen management completely surprise the test team with the external relationship, with predictable results on morale and trust.

Regardless of whether you are involved in the contract negotiations up front or only become aware of the contract afterward, you will need to be aware of certain relevant details in it. As mentioned, elements of the contract often have implications for testing, which you need to understand. In addition, you need to at least be aware of, and plan for, those technical and methodological clauses that will affect testing and quality.

While there is no industry-standard contract for these relationships, there are some typical test-related elements of these contracts that you should be aware of and plan to handle. The first of these involve service-level agreements (SLAs). When the third party is developing all or some of the software product, the SLAs may include turnaround time for defect fixes, which can vary depending on the severity or priority. When the third party is testing all or some of the software product, turnaround time might relate to confirmation testing of defect fixes. In all cases, contracts should specify response time for questions or requests for information. (Requests for proposals relating to additional work usually don’t require an SLA, but we have seen situations in which the customer had to chase the provider around, waving money at them, to get a quote, which is usually a sign that you picked the wrong service provider!) Support staff should be named for the various kinds of questions and information flows that will occur, and methods of contact should be specified. Ideally, there should also be a process that can be used to escalate a question, concern, or problem to senior vendor management if the use of normal channels does not result in timely resolution.

Another important test-related element of these contracts concerns the deliverables. Whichever party does the testing, a clear understanding is needed of what is to be delivered; which of those deliverables are to be tested by the delivering party, the receiving party, or both parties; whether aids to testing such as test data, test cases, test tools, and testing frameworks are deliverables; what information about the deliverables will be provided along with the deliverables—and in what format; and the schedule for those deliverables. Deliverables can include documentation such as user guides, help files, and specifications. Especially if your team is to do the testing, you’ll need test release or test build schedules. Information about deliverables should include a list of any known defects along with release notes that indicate what areas of the system might not be ready for testing yet. Test aids should be thoroughly described, and if test data is to be provided, the requirements for removal of sensitive information (e.g., via anonymization) must be described. The schedule for deliverables should be part of the SLAs, especially for builds that resolve defects.

You might be tempted, or be told by others on the project, to relax and assume problems will work themselves out. But ambiguity in these details about deliverables can be the source of grief on many projects and contracts. It’s hard to be precise in these details, so it’s a common mistake to assume that the fuzzy spots will resolve during the term of the contract. Yes, those fuzzy spots will resolve, but they usually don’t resolve in a way that makes sense or meets the needs of both parties. Our rule of thumb: Be prepared to be precise about deliverables, or be prepared to be disappointed by deliverables.

Yet another important test-related element of these contracts is the quality process used to assure the quality level of the deliverables, which we consider important enough to break out from the deliverables themselves. It’s not enough for the contract to state that deliverables will be of the highest quality. We have seen such vague statements in contracts, in some cases contracts involving millions of dollars. Ideally, the contract should specify expectations for the following quality assurance activities: code reviews; static testing of code and other deliverables; code coverage during unit testing; the approach to unit testing, component integration testing, system testing, and system integration testing and how the test results for those test levels will reported; what constitutes acceptable testing for those test levels, including the additional measures of coverage applicable to each test level (e.g., requirements coverage instead of or in addition to code coverage); and how many residual defects, blocked or unexecuted test cases, and uncovered areas may remain when a deliverable is considered done. Service-level agreements should address many of these areas as well. For example, it’s not enough to say that each unit must be unit tested, with 100 percent statement and decision coverage achieved, if you don’t also say that this must occur before the unit is delivered.

Are we being picky to call out so many expectations about quality and the quality process? You bet. Why are we so picky? Because in most of these contracts there is what’s called producer’s risk and consumer’s risk. The producer’s risk is that the consumer might reject a deliverable that should have been accepted. The consumer’s risk is that the consumer might accept a deliverable that should have been rejected. Without clear expectations about what is acceptable, these risks become unmanageable for both parties. The situation becomes one of politics, a matter of relationships, presentation, and persuasion, rather than of facts.

In addition to the items mentioned in the syllabus, we have an additional suggestion, based on our own experiences with external resources (on both sides of the relationship). We recommend that you look for the contract to include named resources, including human resources with specific skills, each subject to being interviewed and accepted prior to involvement on the project. This clause is not unusual in such contracts, but it is often forgotten. This means that the clients who forget to include this clause will get whichever resources are left over. In the case of loss, delay, or turnover of resources, include a clause about clawbacks, which means that the external party must replace the lost resource in such a way that ramp-up costs or other associated inefficiencies are absorbed by the external party.

Will you be likely to see contracts that address all these elements? Our experience is that you will not. You can work to improve the contracts if you are involved before the contract is signed. You can bring up the questions and problems if you are involved after the contract is signed. The most likely situation is that not all of these elements have been addressed. These unaddressed elements are project risks, and some of these risks affect testing. You should be sure to manage these risks in your test effort if they remain.

Furthermore, depending on the contract, the industry, the product being developed, and the relationship with the third-party, additional items may be required. For example, a third-party that is contracting to produce a medical device probably will be required to obtain levels of government approvals as well (e.g., the FDA in the United States). Similarly, a third-party testing organization that has an existing relationship with the contracting company may require less definition of the SLAs and deliverables, assuming that everyone is happy and comfortable with the established vendor practices and further assuming that those vendor practices are consistently followed.

If you are involved in the vendor selection process, you can use the preceding information to review and contribute to the contract. Check for coverage of each of these items. If you see the contract only after it’s signed, then, as mentioned earlier, be sure to manage the relevant project risks. In addition, gather metrics on how problems with the contract affect testing. You might have been left out of the discussion about the current contract, but such metrics can help you make the argument that you should be involved in discussions about future contracts. Armed with such metrics, you should be able to help the organization improve future contracts. You should also be able to participate in a fair evaluation of whether a given third-party should be used again.

3.4Communication Strategies

As mentioned, communication is one of the main challenges that arises when organizations involve external parties in software projects. Proper communication requires an appropriate approach on a number of dimensions, appropriate tool support, and appropriate attention to other relevant considerations. Let’s examine these issues.

3.4.1Dimensions of Information Strategies

First, the appropriate amount of information needs to be communicated. Each party needs enough information to determine the current status and the appropriate control actions to take (if any). However, there should not be so much information that one side or both cannot process the information provided. Overcommunication can create a feeling of being overwhelmed, while under-communication can create a feeling of being in the dark. Either way, one party (or both) may find themselves surprised by sudden developments or problems, even though the information was available to foresee the situation. If there is consistently too much or not enough information, there may simply be a failure to understand the information needs of the recipients. However, withholding information or providing too much is a tactic sometimes used to obfuscate troublesome situations, so expect problems if a sudden change into overcommunication or undercommunication from the external party occurs, especially if attempts to resolve the problem are met with irrelevant, nonresponsive, or absent replies.

Second, not only is the volume of information important, but level of detail is too. For individual contributors and line managers, insufficient detail can leave the recipients unable to determine status and to decide what to do next. For senior managers and executives, too much detail, without a clear summary, can lead to the wrong conclusions being drawn and the wrong control actions selected. As with the amount of information, problems here can reflect either an intentional obfuscation or simply an inability to understand what the recipients need.

Third, and closely related to the problems with the amount or detail of information, is the frequency of communication. Think of it this way: If you check your watch every second, are you better informed of the current time or distracted from acting on that information? Each party needs information often enough to understand what has happened, what is happening, and what should happen next, but they also need a break in the communication that allows the analysis of that information to translate into insight, decisions, and action.²

Finally, consider the proper style, format, and formality of communication. In some cases, a quick informal email to various members of the project is perfectly acceptable, such as a status update that shows work that is on track to meet the current schedule. In other cases—e.g., missing a major project deadline or failing to provide a key deliverable—such a mode of communication would be seen (at best) as flippant and inappropriate. When troublesome information that requires discussion and tact must be communicated, an in-person meeting or at least a conference call is much more appropriate than an email or voicemail. Regardless of the type of news—good or bad—project status information should almost never be communicated in public (e.g., via Twitter, Facebook, Google+, or on a publicly accessible wiki), except as part of a deliberate, appropriately managed public relations press release. In addition, the manner of communicating with project colleagues is often less formal than the manner of communicating with the other party’s corporate counsel.

The appropriate fashion for handling these various dimensions cannot be uniformly prescribed. The nature of the product will influence communication. For example, in the case of regulated software such as medical or financial software, laws and standards can specify certain communications that must occur and indeed the exact format and style of these communications.

Communication is also affected by the third party involved. Some third parties will have sophisticated existing communication facilities, such as online meeting capabilities, intranets, videoconferencing, or even telepresence systems. Other third parties will tend to rely on simpler systems such as teleconferences, FTP servers, and email. Some third parties will automatically name primary contacts as part of the engagement, while others might have multiple contacts defined based on the particular subject being addressed. Your organization may choose to accept the default way in which the third party wants to communicate or may want to customize the approach.

You need to consider the locations of the various parties involved too. When common time zones and team languages exist, it’s easier to have frequent spontaneous and scheduled communications. If work is proceeding in a number of different locations, this can be difficult. When only designated points of contact within the third party speak the other organization’s project language, the pathways for communication are more limited.

The criticality and complexity of the project influences the communication. Generally, as project criticality or complexity increases, the project team needs to increase the frequency of communications, the number of prearranged contingency communication channels (in case of problems), and the precision, reliability, and alacrity of the escalation processes. For simple projects, especially when the third party’s deliverables are not on the critical path for the project, you can rely on regular status reporting and deliveries, often via email or other simple channels.

Finally, any previous working relationships with the third party and the points of contact will influence communication. Established ways of communicating will often be continued, simply because it’s simpler. Rex recalls that in one project, he had an informal—and sometimes confidential—communication channel with a vendor’s test manager. He would give Rex frank appraisals of quality and warn him when problems were arising.

The test manager must be aware of each of these factors before defining the communication strategy for the project. For example, if the project involves having the development and unit testing done by an offshore facility, it might make sense to require weekly detailed status reports, daily defect reports, semi-weekly conference calls, and perhaps even an agreement on the language to be used in written communication.

Ideally, routine communication should be addressed in the contract. However, exceptional situations—which, in our experience, tend to occur more than once on a project—are not easily codified in a contract. The way a party in such an exceptional situation chooses to communicate with the other party speaks volumes. There is an aphorism that says that one’s character is defined by their actions when no one else is looking. That is true, but it’s equally true that one’s character is defined by their actions, and their communications, when everyone is looking, and looking very hard. Certainly, when problems arise—and don’t they always?—we respect people who act and communicate plainly, honestly, transparently, and with an eye toward the ultimate success of the larger goals, especially if they step forward to take responsibility for whatever problems have occurred.

3.4.2Communication Tools

In our experience on projects that involve third parties, testing—whichever party is doing the testing—goes much more smoothly when appropriate tools are used for defect management and test management. By appropriate we mean two things:

• The tools were actually designed for the purposes of defect management and test management, respectively. We have yet to find a client that was fully successful in forcing a tool to manage defects, tests, and/or coverage when the tool wasn’t designed for that purpose.
• The tools represent the best of breed in terms of tools designed for these purposes. The tools can be commercial or open source—we have clients successfully using both types of tools—but the tools should be carefully selected according to best practices for tool selection.

It is said that the craftsman is known by his (or her) tools. Indeed, you can drive nails in the wall by banging on them with the handle of a screwdriver, but it’s certainly not the right way to do so. The use of the right tools will facilitate effective, efficient, and consistent communication between the various parties as well as gathering project, process, and product metrics. (We’ll discuss metrics further in Chapters 6 and 8.)

Not only must we have the right tools, but we must have a single set of tools across the project or program. Trust us on this: We have tried to integrate test results data from across disparate repositories. It’s a nightmare, and the resulting information is considerably less reliable than it would be with a unified set of tools. Yes, establishing appropriate access, and especially support, for these tools can be hard work. However, remember that the deliverables of testing are forms of information, and these tools are the conduits by which that information is transferred between the parties.

It’s not just enough to give everyone access to these information tools. There are two other important factors: motivation and education. The contract should be defined such that the third party is motivated to provide accurate, timely, and credible information using these tools. That motivation should flow from the managers who signed contracts to the individual contributors, but it won’t always do so. Be sure the contract includes penalties for bad data in these tracking systems, and don’t be shy about escalating data problems when they occur.

In terms of education, there should be at least usage guidelines for the various narrative and classification fields in the tools. The best practice is to have mandatory training that ensures that each party agrees on the proper use of such tools. This includes tactical information such as severity and priority classifications as well as more strategic information such as where defects are introduced, detected, and removed. One of our clients learned this lesson the hard way when they discovered that some 10 to 25 percent of their technology budget was being wasted on excess costs of failure. Worse yet, their defect information was so full of improper classifications that they could not determine the best way to attack that waste.

3.4.3Other Communication Considerations

While we’ve been focused on communications between testers and developers in this discussion of tools, there are other external stakeholders who have communication needs. You might find that you need to communicate with end users, regulatory authorities, and other stakeholders. These stakeholders probably don’t want to be embroiled in the daily tactical details, but they might need to understand the test basis (i.e., the test conditions to be covered) and the high-level test results. Some of our clients open these communication channels by using a risk-based testing strategy and including these stakeholders in the risk identification and assessment processes.

Some of the communication can be done via tools, but not all of it. Meetings happen, and often meetings (or just plain phone calls between two people) are the most efficient and rapid way to resolve open issues. When these meetings must occur across time zones, as they often will when third parties are involved, the best practice is to establish regular time slots that accommodate the different parties. It’s not just considerate to take people’s schedules into account, but you can hardly expect someone’s best work from them if they must be in a meeting at midnight or worse yet 3 a.m.

Another topic relevant to communication is that of holiday and vacation schedules. In most cases, people will be highly resistant to having to engage in anything but emergency communications while on a holiday or vacation. This consideration is especially important when you have teams working in distributed sites, often in different countries, sometimes with very different holidays, religions, and cultures. You might also find, if you are from North America and East Asia, that the work culture in other parts of the world can be less intense and fast-paced than you have come to expect. You might find a more relaxed pace refreshing or frustrating, but either way the holiday and vacation schedules are generally not subject to negotiation or override-by-dictate, so you must accommodate.

If you are working in an organization following an Agile approach, there should be daily meetings between the members of the projects. These are sometimes called “stand up” meetings (because they are supposed to be kept brief by keeping people standing). When the work is distributed across time zones, they might be sprint handoff meetings. Whatever the name and whenever they happen, Agile methodologies require frequent and regular communication, due to the reduced amount of documentation available. Everyone, including participants distributed around the world, must be able to participate in these meetings.

3.5Integrating from External Sources

It’s not unusual for a project to involve integrating components or even entire systems from external parties. There is some amount of testing required by the recipient party under any circumstances, but the exact amount of testing, and the levels of testing, can vary. Let’s look at various situations that can exist here.

At one end of the spectrum, a third party or multiple third parties deliver a fully developed and tested product. This is sometimes called a turnkey project. In this case, the developing party (or parties) should have completely tested the product. Your test team should plan and execute an acceptance test. The basis of the acceptance test should be the predefined acceptance criteria from the contract, from requirements specifications, and from other sources of information about what the vendor was obligated to do. The objectives of this testing should not include detecting defects because the third party should have sufficiently tested the product prior to delivery. That said, it’s not unusual for third parties to deliver products that were clearly not sufficiently tested. Contracts should be structured so that the third party pays a substantial penalty for defects detected during acceptance testing.

It’s often the case that these externally developed systems must be integrated into a larger set of systems. A number of our clients sell enterprise software that must work in data centers with many other systems, some externally developed and some internally developed. Another similar situation arises when a product is integrated with other products and then released as a complete package. Either way, system integration testing is a key part of completing these deliveries. Unlike with acceptance testing, it’s not always possible to anticipate the various test conditions that will arise, so some number of defects will typically be found. Testing should be planned to accommodate those defects, but at the same time the contract should reflect what a reasonable number of such integration defects would be.

Moving along the spectrum of possible test involvement, we come to the situation where the third party delivers a component they have developed and unit tested. The wise test manager will request proof of testing, including code coverage metrics, the unit tests themselves (ideally automated and executable), and the test results. Assuming that the test manager is satisfied with the testing, acceptance testing of the component should occur, possibly using (in part) the unit tests provided by the third party. (As mentioned earlier, the contract should clearly specify what is required in terms of acceptance.) Following successful acceptance of the component, the test team should proceed to component integration testing, system testing, and perhaps system integration testing. This component could be integrated with other externally developed components, where each of the components would follow the same process specified in this paragraph, or with components developed internally. Acceptance testing often does not occur for internally developed components—though it does on properly run Agile projects—and doing so, regardless of lifecycle, is a good software engineering practice.

A related situation occurs when the third party delivers a developed product but cannot prove that it was tested or frankly admits that it was not tested sufficiently or even at all. Just to state the obvious, this situation really should never happen, and the contract should absolutely prohibit the possibility. However, sadly, this is not uncommon and in fact might be the most common situation. What to do? In the ideal case—well, the ideal recovery case, because this situation is far, far from ideal—your team would be able to conduct the unit tests. That typically involves technical capabilities that your team might not have, so you might have to proceed with component integration, system, and system integration testing. If there’s any way for your team to perform an acceptance test of the component, by all means do so, but it might not be independently testable by your team and the developers might well have better things to do. If you must proceed with higher levels of testing without any sort of proof of testing or acceptance testing, you should carefully track the costs of poor quality associated with this situation. Also, if there are contractual penalties based on code quality and defects found, you should also carefully document test escapes.

At the entire other end of the spectrum, what of the situation in which a third party tests an internally developed product—that is, when some or perhaps all of the testing is outsourced for a project that was developed internally? Here, the test team is not necessarily involved, but the test manager should be. The test manager should be able to define, or at least accept, the test strategy to be used. The best practice would be for the test manager to meet with their counterpart in the third party to review and agree on the test approach, the test documentation, the test tools, the test automation (if any), and the tracking of the test results. The requirements for delivering these items should also be discussed. Furthermore, after this discussion, the best practice is that the test manager remains engaged with their counterpart in the third party to monitor the status of testing as it proceeds.

While considering these various types of external party situations, also consider the level of integration and handoff documentation required. For example, if you are going to perform component integration and/or system integration testing, you’ll need to know how things fit together. As another example, when we mentioned earlier that you should ask for “proof of testing,” well, what exactly would constitute adequate proof, sufficient for you? What requirements exist for your project? What are the various milestones that will occur, and how can you measure adequacy of testing and quality at those milestones?

As important as testing and quality metrics are in any situation (as discussed in Chapters 6 and 8), in situations with external parties, they are even more important. Ideally, these metrics are part of how the external party’s effectiveness and efficiency are measured, and the test manager is involved in defining and monitoring the metrics. The metrics should include test escapes, defects trends, unit test coverage, and so forth.

As mentioned in the section on contractual issues, the best practice is that issues such as the ones discussed in this section are firmly defined in, and enforceable through, the contract. Without the contract behind you, no matter how egregious the violation of testing or quality best practices, you will often find that there’s little you can do to solve the problem for the current project. However, that doesn’t mean you can’t change future projects. These metrics can drive such changes.

3.6Merging Test Strategies

As you might have gathered from this chapter so far, the involvement of third parties poses a number of challenges for test management. It’s especially true when some or all of the testing is done by the third party. That might seem counterintuitive, but it’s often the case that test managers or test directors retain ultimate accountability for the testing, even when that work is not done by their team. The test manager must assemble a coherent picture of testing and quality from across multiple locations, multiple organizations, or multiple internal or external groups.

It can feel like trying to assemble a realistic Goya painting such as The Duchess of Alba from a cubist Picasso painting such as Dora Maar au Chat. After all, both are portraits, paintings of an individual person with a pet. Each painting is of a woman, indeed of attractive women of the same approximate age, and each woman has the same general Western European features, so how hard could it be? However, go online, look at these two paintings, and figure out how to manage it. Cutting the Picasso painting into pieces and trying to make the Goya painting from those pieces will clearly not work. The answer is to try to paint a coherent Goya from the beginning.

Let’s look at the various ways in which third parties can complicate the situation and how those complications can be resolved. One of the first avenues of complication is the merging of disparate defect and test management information into a single tool. The potential problems here are myriad, and some were mentioned previously. It’s difficult to ensure consistent classification and narrative information across disparate tools unless strenuous and ongoing efforts are taken to harmonize the various fields and the way they are used. One example of this type of inconsistency is when one party prioritizes defects on a scale from 1 to 5 (where 1 is the highest priority and 5 is the lowest) but the other party’s scale is reversed (5 is highest priority and 1 is the lowest). Even after all parties agree on a prioritization scheme, care must be taken to make sure there is no confusion in how to report defects going forward and how to manage the defects that are already in the systems. The lack of consistency, especially in the classifications, inserts noise into the test results, making it difficult to understand the process, product, and project implications. Avoid this situation if at all possible. If disparate defect and test management tools must be used, invest the effort to harmonize the information gathered. Simply because the immediate tactical needs of the project are met doesn’t mean that you can obtain adequate process information in the long term.

Beyond information management tools, there are other test automation strategies and tools involved. For example, if the third party creates automated tests using Selenium and your team uses Rational Functional Tester, you will have problems integrating the automated tests and even making sense of the results across the set of tests. When there is no common definition of a test—e.g., how many conditions are covered, how many inputs are submitted, how many outputs are confirmed, etc.—it’s not possible to aggregate test counts to report test status. If your third-party partner sends you tests implemented with J-Test and you are using an open source tool like J-Unit, you will have trouble using those tests.

And this leaves open the wider question of whether the automation strategies are the same. If your strategy is to automate regression tests only and your partner’s strategy is to automate reliability, performance, and regression tests, then your organizations are not aligned in terms of strategy. If you want automated functional regression tests at the graphical user interface for every feature before that feature is released but your partner intends to deliver regression tests at the application programming interface (API) only, then your organizations are not aligned in terms of strategy.

Another area of wide variance between test organizations is the definition of test levels. While the ISTQB program has clear definitions for unit, component integration, system, system integration, and acceptance test levels, these definitions are not universally accepted or followed. If you can standardize with your partner on these levels of testing—or any other mutually agreed-upon set of levels—then you can proceed to define clear responsibilities for each level. Otherwise, gaps and overlap in test coverage will tend to arise and persist.

In addition to defining the test levels and ownership of those levels, we need to make sure that commonly accepted entry and exit criteria exist across those levels. What does it mean for us to be ready to start one level? What does it mean for us to be ready to declare another level complete? Are the definitions of entry and exit criteria consistent between adjacent levels of testing? Without clarity on these questions, testing activities cannot be coordinated across teams.

With clearly defined levels, ownership, and touch points between each team, we can identify a clear plan for who does what testing, when and how they do it, and with what deliverables. The idea is that we have clearly defined activities, with overlap only where we want it. A lack of defined activities can lead to gaps in our coverage, which means less testing and lower quality, while excessive overlap means efficiency is reduced and the meaning of the test results is confused.

In the Abrahamic religions, there is a story of the Tower of Babel. In this story, people set out to build a tower to Heaven so that they could discuss the whys and wherefores of the human condition with God. God frustrates their plan by telling the angels to “confuse their tongues,” causing them to speak different languages. As often occurs when people cannot communicate on a project, the tower project failed. Whether you are building a tower (or just a stairway) to Heaven or a banking application, you will need a common glossary of project terminology. This should cover both technical and testing terms.

With mutually defined test levels, responsibilities, criteria for those levels, and terminology, the next step is to define shared testing and quality objectives. How is coverage to be measured at each test level, and what constitutes adequate coverage? What quality metrics should be used, and what constitutes adequate quality for release? With consistent objectives, you can define metrics and reporting frequencies for each team involved in the project. Only with consistent answers for these questions, across all the partners, can the quality of a release be managed in a truly meaningful and informed fashion.

As discussed in Chapter 1, there are various test strategies that can be used. It’s not necessary—or even ideal—for every partner involved in testing to use the same strategy. Strategies can complement each other. Ideally, a conversation occurs where the various parties involved in testing discuss the different test strategies and arrive at a common understanding of which party will employ which strategy and when. The overall result of testing should meet the needs of the project through a blend of test strategies across the various partners. The test manager for the procuring organization (i.e., the client) should be able to ensure that the disparate test strategies align with their organization’s overall test strategy.

When the test strategy includes risk-based testing, then the risk analysis should be comprehensive, across all the partners involved. The partners might or might not be involved in doing the risk analysis, but they should certainly be aware of the results and how their testing addresses the identified and assessed risks. This provides a centripetal force that aligns the testing effort across the various partners.

When merging test efforts with third parties, test managers should not expect the complications discussed in this section to resolve themselves. They won’t. Instead, as the project progresses, the damage created by these complications will increase. So, before the project starts, the test manager should spend some time planning and coordinating these areas. Even if the project manager—and perhaps you—forgot to include this work in the estimate, you should do it anyway. The cost of not doing it will exceed the cost of doing it, and the cost of not doing it will certainly not fit within the project estimate.

This caveat applies to the division of test work in general. You need to agree on these arrangements before the project starts or problems will occur. But what if the project starts and your testing counterparts are not available for planning work? In that case, it will be difficult (or maybe impossible?) to resolve the issues mentioned in this chapter. Ideally, the contract requires that all the appropriate parties are involved in the planning of the testing work before the project begins.

3.7Verifying Quality

Suppose you went to a restaurant for dinner, sat down, and told the waiter, “Bring me dinner and a drink.” You didn’t provide any further details, though you had something specific in mind. What are the chances that you’ll get the dinner and drink you expected? While no one would ever do this in a restaurant, it happens sometimes on projects that involve third parties.

If we have certain expectations and requirements for an engagement with a third party, those should be defined and clearly communicated between the parties. The best practice is to have that definition and communication before the project starts and to put the agreed-upon terms into the contract. If the third party is delivering software, then these requirements should include quality targets, including measurements of those targets. The measurements should be objective and not subject to distortions, as discussed in Chapter 8.

In addition to defining the requirements, the point at which those requirements must be met should be defined. This can be done by defining entry and exit criteria that establish quality gates for deliverables. Because these quality gates will control the start and end of project phases, they should be synchronized with the phases of the project and aligned with project schedule milestones.

The Expert Test Manager syllabus provides a number of examples of entry and exit criteria for various test levels. We reformatted them into Table 3-1 below and provided our comments and suggestions on implementation or improvement of each criterion.

Table 3-1 Annotated Entry and Exit Criteria

It’s important to note that Table 3-1 provides only a small sample of the criteria you would include. On an actual project, you should have a large and thorough set of criteria, addressing various issues that affect the testing work on the project, the test results, and the quality of the software being tested.

As mentioned in the Foundation syllabus, typical entry criteria for test levels should address issues such as the availability, readiness, completeness, and quality of the test environment; the availability, readiness, completeness, and quality of the test tools, including their installation in the test environment as needed; the availability, readiness, completeness, and quality of test items being delivered for test execution; and the availability, readiness, completeness, and quality of the test data. As mentioned in the Advanced syllabus, entry criteria should also address whether the tests are complete and ready to run; whether the tools are available to support test management, defect tracking, and (if applicable) automated test execution; and whether defined approaches for test results logging and tracking, defect reporting, and test metrics analysis exist and are understood by all the testers.

As mentioned in the Foundation syllabus, typical exit criteria for test levels should address issues such as the level of coverage achieved, in terms of code, functionality, requirements, or risks; predicted numbers of defects remaining, defect density, mean time between failure, or availability; cost of continuing versus ending testing; the residual level of quality risk (in terms of known defects, known failed tests, or gaps in test coverage); and schedule targets.

The stringency and formality of the criteria will vary. The product and application domain influences the criteria; for example, safety-critical systems need tougher criteria than a company’s promotional web page. The past experience—good or bad—with the third party influences the criteria; for example, if a vendor provided poorly tested software in the past, the rigor of the entry criteria should be increased. The requirements for the system being built and those in the contract or agreement influence the criteria; for example, if usability is central to the value of the software, usability testing and its results should be in the criteria.

In addition to criteria to measure the status of the product, there should be well-defined, objective, measurable project milestones that allow test managers, project managers, and product managers to track the testing and the project against a schedule. To make a milestone measurable, it is imperative to have clearly defined deliverables, with a linkage to the entry criteria for those deliverables. While project or product managers should track the overall schedule, test managers should track milestones, or at least participate in the tracking of milestones, that relate to testing work and the quality of the system under test.

As important as these entry criteria, exit criteria, and milestones are for a single colocated team on a project, when other parties are involved, they become essential bulwarks against chaos and disorder. Therefore, it’s important to spend the time to carefully craft the proper criteria and milestones. Not only do all important issues need to be addressed, but the criteria and milestones must be measurable in a way that all sides will agree is objective and conclusive.

It’s frustrating to have criteria and milestones that are contested and reliti-gated by a third party once problems arise with its deliverables. It’s also frustrating to find that a third party is trying to find gaps or ambiguities in criteria. However, those situations can happen. For you to deal with these situations, the criteria and milestones must be complete, measurable, objective, and—here comes the tricky part—enforceable. To be enforceable, the criteria and milestones must be clearly traceable to some clause in the contract, if not actually directly in the contract (which is the better practice). There needs to be a defined process that allows the test manager to work with the third party to resolve a violated criteria or missed milestone and, if resolution proves impossible, a defined process to escalate the problem.

During that period of resolution and, if necessary, escalation, the test manager should have clear direction on how to proceed. If testing is not to start or conclude unless certain entry or exit criteria, respectively, are met, then the test manager must have the authority to effectively stop the project, and the test manager must—absolutely must—be insulated from any negative consequences associated with such an action. Be very careful here; we have seen entire testing groups shipwreck themselves—literally put themselves in such a bad way with their colleagues that the test group ended up being dissolved—by allowing themselves to be dragooned into a process cop role, with rigorous entry and exit criteria that they had to enforce on very unhappy coworkers. We feel the best practice is for the test manager to report the status of the criteria to project and product managers and have them make the decision about whether to enforce or waive the criteria.

3.8Sample Exam Questions

In the following section, you will find sample questions that cover the learning objectives for this chapter. All K5 and K6 learning objectives are covered with one or more essay questions, while each K2, K3, and K4 learning objective is covered with a single multiple-choice question. This mirrors the organization of the actual ISTQB exam. The number of the covered learning objective(s) is provided for each question, to aid in traceability. The learning objective number will not be provided on the actual exam.

The content of all of your responses to essay questions will be marked in terms of the accuracy, completeness, and relevance of the ideas expressed. The form of your answer will be evaluated in terms of clarity, organization, correct mechanics (spelling, punctuation, grammar, capitalization), and legibility.

Question 1

LO 4.6.1

Assume you are managing a testing project for an e-commerce website. A third-party testing service provider, using its own test environments, will handle the performance testing of the site during system testing. You will deliver the test object (i.e., test items used to install the website in its test environment) to the testing service provider once system testing has started. Which of the following is the most important element in aligning test strategies in this situation?

1. Performing a comprehensive assessment of the testing service provider.
2. Defining a unified, compatible set of test tools.
3. Agreeing on the system test exit criteria with the provider.
4. Defining testing and product performance goals.

Question 2

LO 4.3.1 and LO 4.4.1

Consider Scenario 2. Further assume that you have determined that it is most economical to use an outsource testing service provider to perform compatibility testing with various mobile devices, browsers, operating systems, and other platform factors that affect the way in which your games work. Management has asked you to locate a competent provider to carry out this work.

You plan to have the provider do the compatibility testing as part of the system test level. Your team will continue to manage the system testing and carry out the rest of the functional and non-functional testing.

Which of the following statements best summarizes an element of effective, timely communication of test results by the vendor that you should include in the request for proposals (RFP) you will use to locate and select your testing service provider?

1. Require the testing service provider to send daily test results reports via its existing tools to a designated point of contact in your team.
2. Require the testing service provider to report its test results, once all testing is completed, directly into your test and defect management systems.
3. Require the testing service provider to report its test results immediately and directly into your test and defect management systems.
4. Do not include any requirements on test results reporting in the RFP because too many qualified vendors will be excluded from bidding.

Question 3

LO 4.2.1 and LO 4.5.1

Consider Scenario 2. Further assume that your company is considering an outsource development vendor with which no current or past business relationship exists—that is, it will be a new vendor to your company. Your company’s objective is to outsource development of a complete product that will complement the existing line of games but is not considered essential to the business. The managers negotiating with the vendor want your input on planning the testing and putting the proper test-related clauses in the contract.

Which of the following statements best summarizes the advice you should give management in terms of planning the testing responsibilities and contractual arrangements in this third-party relationship?

1. Plan a cursory acceptance test of the delivered product, and contractually require that the vendor’s testers have ISTQB Foundation and Advanced certifications.
2. Plan a thorough system test of the delivered product against clearly defined contractual requirements, with contractual rewards and penalties based on the results of this system test.
3. Plan a thorough acceptance test of the delivered product, but also plan to audit, and if necessary, manage the vendor’s unit test, integration test, and system test process and results throughout the project.
4. Plan a thorough acceptance test of the delivered product against clearly defined contractual requirements, with contractual rewards and penalties based on the results of this acceptance test.

Question 4

LO 4.7.1

Consider Scenario 1.

Part 1: Create a set of entrance criteria for the acceptance test.

Part 2: Create a set of exit criteria for the acceptance test.

1Rex has written about distributed testing in various books and articles, but most especially in Managing the Testing Process: Practical Tools and Techniques for Managing Hardware and Software Testing/ Edition 3, Chapter 10.

2For a timely discussion about how excessive, continuous distraction has become a problem for everyone, not just those of us in technology, see Nicholas Carr’s interesting book The Shallows: What the Internet Is Doing to Our Brains. Ironically, Rex downloaded this book over the Internet from Amazon.com and read it on his Kindle, but at least he didn’t stop reading every other paragraph to check his email.