A testing technique is nothing until it leaves home and makes a name for itself in the real world. It is one thing to talk of adventure and travel, and quite another to live it. This chapter was written by the first testers to apply the tourism metaphor purposefully on real projects and under real ship pressure.
The in-the-small and especially in-the-large techniques were born within the Developer Division of Microsoft and saw their first use within our teams in Redmond, Washington, and our India Development Center at the hands of David Gorena Elizondo and Anutthara Bharadwaj, respectively. They saw their first public unveiling at EuroSTAR 2008 in Den Haag, Netherlands,1 in November 2008. Since then, I am personally aware of some dozens of groups outside Microsoft who have made them work.
1 You can find information about EuroSTAR at http://qualtechconferences.arobis.com/content.asp?id=91.
Later that same month, two efforts were launched inside Microsoft to take the tours “on tour.” A formal effort within Visual Studio was started that is still ongoing at the time of this writing. At the same time, a broader grassroots effort began companywide. I began it by sending an email to the Test Managers and Test Architects in the company, those testers of the highest level and broadest reach within Microsoft, asking to be referred to talented testers, regardless of their experience. I specifically asked for promising manual testing talent.
I was flooded with responses and whittled down the pool with personal interviews, frankly by choosing those who seemed most enthusiastic. I like working with passionate people! Then we began training, and each tester read and edited this text. After that, we began touring.
Many product lines were involved, from games to enterprise, mobile to cloud, operating system to web services. I’ve selected five of the most informative to appear here. The remainder are still ongoing, so don’t be surprised if there are follow-ups on Microsoft blogs or elsewhere.
As you will see in this discussion, many of the tours survived being put into practice intact and were applied exactly how they were documented in Chapter 4, “Exploratory Testing in the Large.” However, many variations were crafted on-the-fly, and in some cases new tours were created from scratch. Not only is this acceptable, it’s desirable. I expect any team to find tours that work for them and tours that don’t. That’s why this chapter is here: to show how the tours work in practice.
The experience reports that follow are from the following Microsoft testers and appear with their permission:
• Nicole Haugen, Test Lead, Dynamics AX Client Product Team
• David Gorena Elizondo, SDET, Visual Studio Team Test
• Shawn Brown, Senior Test Lead, Windows Mobile
• Bola Agbonile, Software Development Engineer in Test, Windows
• Geoff Staneff, SDET, Visual Studio Team System
By Nicole Haugen
My team is responsible for testing Dynamics AX’s client. Dynamics AX is an enterprise resource planning, or ERP, solution that was implemented more than 20 years ago in native C++ and was acquired when Microsoft purchased Navision. As the client team, we are considered to be a “foundation” team that is responsible for providing forms, controls, and shell functionality that the rest of the application is built on. Prior to this, my team was primarily testing public APIs, so Dynamics AX was a mind shift to testing through a GUI. When we made this transition, we learned several things:
• Many of the bugs that we found were not being caught by the test cases that we had identified in our test designs.
• Testing through the GUI introduced a seemingly infinite number of scenarios and complex user interactions that were not easily captured using automated tests.
• Whether a test is automated or manual, it is a regression test that must be maintained. My team has thousands of tests, so we must constantly consider the return on investment associated with adding a new test case to our regressions.
• Dynamics AX is a massive application and there was a lot about it we did not know, let alone how it should be tested.
Exploratory testing helped us to address all of the proceding issues. As a result, we have incorporated it into our process in the following ways:
• Before each feature is checked in, a tester performs exploratory testing on the code; this is to find important bugs fast and preferably, before they are ever checked in. We also follow this same practice for check-ins related to fixes of critical or high-risk bugs.
• Exploratory testing is used to help with the development of test cases while writing our test designs. It helps us to discover new scenarios that may have been missed in requirements.
• During our manual test pass, we use the test scripts as a jumping-off point to inject exploratory testing, as described in Chapter 5, “Hybrid Exploratory Testing Techniques.” It has been my personal experience that the manual tests as they are written rarely detect new issues; however, with even the slightest detour from the test, many bugs are found.
• During bug bashes, we perform exploratory testing, which has helped lead us to investigate other areas outside of the current feature area to discover related issues.
The concept of tours makes exploratory testing much more concrete, teachable, and repeatable for us. This section describes a few tours that have been particularly useful for finding bugs in Dynamics AX.
When traveling by mass public transportation, there is always risk that travelers may board the wrong route or get off at the wrong stop. Another drawback is that it is often impossible to take a direct route to a desired location. In the rare case that it is, the exact same route is usually taken over and over, with no variety to offer those that have visited the destination more than once. One surefire alternative is to travel by taxicab. While the fare of a taxicab costs more, the old adage “you get what you pay for” definitely applies. In a city such as London, where there are more than 25,000 streets, cab drivers must take rigorous exams to ensure that they know every route possible to get from point A to point B within the city. You can bet your bottom dollar (or should I say pound) that cab drivers know which route has the shortest distance, which offers the shortest amount of travel time, and even which is the most scenic. Furthermore, each and every time, passengers will consistently arrive at their specified location.
This same type of tour is also applicable to testing software applications. To reach a desired screen, dialog, or some other piece of functionality, there are often myriad routes that the user can take. As a result, testers have the same responsibility as taxicab drivers, in that they must educate themselves on every possible route to a specified location. Testers can then leverage this knowledge to verify that each route consistently delivers users to their target destination. In some situations, the state of the target destination may be expected to vary depending on the route, which should also be verified. Notice that this tour is a derivative of the Obsessive-Compulsive tour, in that the ultimate goal is to repeat a specific action; however, rather than exercising the exact same path to the action over and over, the key difference is that this tour concentrates on exercising different paths.
Consider an example using Microsoft Office’s Print window. To open this window, users have various options:
• They can send the Ctrl+P hotkey.
• They can select the Print menu item through the Office menu button.
• They can click the Print button on the toolbar of the Print Preview window.
No matter which of these routes the user chooses, the end result should be the same: The Print window opens.
Conversely, there is also the concept of the Blockaded Taxicab tour. The objective of this tour is to verify that a user is consistently blocked from a destination regardless of the route taken. There are many different reasons why a user may be prevented from accessing functionality within an application, whether it is because the user does not have sufficient permissions or it is to circumvent the application from entering into an invalid state. Regardless, it is important to test each and every route because it is surprising how many times a developer overlooks ones.
Although the preceding example involving the Print window is becoming somewhat contrived at this point, let’s continue with it for the sake of simplicity. Suppose that a user should be prohibited from printing hard copies. Because we know that there are many different ways to access the Print window, it is important to verify that no matter how the user tries to access this window that the user is consistently prevented. This means that at a minimum, the Ctrl+P hotkey should result in a no-op and that the Print menu item and toolbar button should become disabled.
One great aspect of London is the sheer diversity of people that it encompasses. Without leaving the city, let alone the country, a tourist can experience cultures from around the world. For example, a tourist might choose to visit London’s Chinatown, where the delicious aroma of Chinese cuisine can be smelled and traditional Chinese scripture can be viewed throughout. Similarly, a tourist may opt for an Indian restaurant tucked within one of London’s busy streets, where diners enjoy smoking tobacco out of hookahs. Tourists can choose to submerse themselves into many different cultures as an exciting and wonderful way to spend a vacation in London.
The Multicultural tour applies to testing because it is important that testers consider the implications of providing software that is localized to different countries around the world. It is essential that language, currency, formatting of dates, types of calendars, and so forth be adapted appropriately to the end users’ region. In addition, it is important that functionality continues to work as expected, regardless of the locale.
Although testing a product’s localization can be very complex, here are a few basic ideas to get you started. Notice that you do not necessarily need to be fluent in a different language to perform these types of tests:
• A basic aspect of localization is that no text should be hard-coded (and thereby prohibiting it from being translated to the appropriate language). A great way to test this is simply to change the application’s and operating system’s language and verify that labels, exception messages, tooltips, menu items, window captions, and so on no longer appear in English. Also, it is likely that there are specific words that should not be translated, which should also be verified, such as words that are part of a brand name.
• Try launching the application under test in a right-to-left language, such as Arabic; verify that controls and windows behave correctly. With right-to-left languages, it is interesting to change the size of windows and make sure that they repaint correctly. Also, test controls, especially custom-implemented controls, and make sure that they still function as they did in left-to-right mode.
This list is obviously far from being comprehensive but at least gives an idea of general aspects of an application to verify without necessarily getting language-specific.
This section is dedicated to the bugs that have been collected as souvenirs while traveling throughout Dynamics AX using the Taxicab and Multicultural tours. While these tours have helped to find many bugs, here are a few of my favorites.
Dynamics AX has a known limitation: Only eight instances of the application’s workspace can be opened simultaneously.2 Any more than eight will cause the entire application to crash (which, by the way, is an issue that could have been caught by using the Test One, Get One Free tour). Because of the complexity involved with fixing this issue, it was decided that users should simply be prevented from opening more than eight workspaces so that they never get into the situation where the application crashes.
2 Note that although multiple application workspaces are created, in this scenario they are tied to a single Ax32.exe process.
When I learned of this behavior, the Taxicab tour immediately came to mind. Specifically, I began to think of all the possible ways that a user can open up a new workspace. Like any experienced taxicab driver, I came up with several routes:
• Clicking the New Workspace button on the Dynamics AX toolbar
• Sending the Ctrl+W hotkey
• Executing the New Workspace menu item under the Dynamics AX Windows menu
• Clicking the New Workspace button that exists in the Dynamics AX Select Company Accounts form
Once I was equipped with all the possible routes, I then proceeded to open seven application workspaces. With seven workspaces open, my first objective was to verify that an eighth workspace could be opened using each of the routes. As it turned out, each route was successful in doing so.
Next I applied the Blockaded Taxicab tour. Now that I had eight workspaces open, my objective was to verify that the user was blocked from opening a ninth workspace. I attempted to travel the first three possible routes, and in each case I was prevented. However, when I got to the fourth route, it was still possible to create a new workspace. As a result, I was able to launch a ninth workspace, which caused the application to crash.
Like most applications, Dynamics AX provides several common menus to the user, such as View, Windows, and Help. To test Dynamics AX menu behavior, I executed each menu item to ensure that the desired action was performed. This, of course, is a pretty straightforward approach; so to make it more interesting, I decided to apply the Taxicab tour. Specifically, I executed menu items by traveling the following routes:
• Clicking the menu and menu item with the mouse
• Sending the hotkey that corresponds to the menu item
• Sending the menu’s access key followed by the menu item’s accelerator key
Sure enough, I discovered a bug when I used the third route (using the access and accelerator keys to execute the menu item). For example, to execute the Help menu’s Help menu item, I attempted to send the Alt+H access key to open the menu, followed by the H accelerator key to execute the menu item. Surprisingly, I did not get that far because the Help menu failed to even open. On the bright side, this was an important accessibility issue that was identified and fixed before the product was shipped.
Dynamics AX is shipped in many different languages, including right-to-left languages, so it is important to verify that the application supports globalization and localizability. The Multicultural tour has helped to discover many bugs in this area.
One bug that I discovered while using the Multicultural tour involves the tooltips that display menu access key information to the user. Consider the Windows menu tooltip, which in English is displayed as Windows <Alt+W>.
I opened Dynamics AX in various languages, such as Italian, and noticed that the Windows menu tooltip displayed Finestre <Alt+W>. Although the name of the Windows menu had been properly translated, the access key had not. Instead, it should have displayed Finestre <Alt+F>.
A majority of Dynamics AX controls are custom, so it’s very interesting to run the application in right-to-left languages and verify that the controls behave correctly. In fact, a great example of a bug found by one of my colleagues involves the Dynamics AX Navigation Pane, which can be put in either an expanded (see Figure 6.1) or collapsed state (see Figure 6.2) by the user:
Notice that the << and >> buttons at the top of the Navigation Pane are used for changing the pane’s state. When my colleague launched Dynamics AX in a right-to-left language, clicking the << arrow button simply failed to collapse the Navigation Pane; however, these buttons worked fine in left-to-right languages.
Both of these bugs would have gone undiscovered if we had not taken the Multicultural tour through Dynamics AX.
While applying the tours described in Chapter 4, I compiled the following list of tips as a “traveling” companion for the savvy tester.
When we are testing a product with a GUI, the Supermodel tour is vital to eliminating glaring flaws in the interface. A useful tip for using this tour to find even the most subtle flaws is to combine it with some of the other tours described in Chapter 4.
While perusing the interface, always take care to look beyond the current window or control that you are primarily focused on to see how the rest of the application appears. This technique is comparable to the Supporting Actor tour, where you must “turn your attention 10 degrees left or right” to get the full effect. For instance, I once found a bug where I opened a pop-up window on a form and the title bar became grayed-out, thereby making it appear as though the entire form had lost focus. While my main attention was on the pop-up window, by taking a step back to look at the entire form, I caught a very subtle bug with the title bar.
The main premise of the Back AlleyMixed Destination tour is to test how different features interact with one another. With respect to a GUI, it’s important to verify how features of the external environment affect the appearance of an application. Here are a few examples:
• Modify the OS display settings, such as to high contrast, and take a Supermodel tour through the product to verify that all controls, icons, and text display properly.
• Use Terminal Services to remote into the machine that has the application installed and verify that there is no painting or flickering issues as windows in the application are drawn.
• Run the application with dual monitors and verify that menus and windows display in the correct monitor.
Because most first impressions are based on looks alone, bugs involving the appearance of an application can lead to perceptions of an unprofessional, poorly engineered product. Unfortunately, these types of bugs are often deemed as low priority to fix. A handful of appearance-related bugs in an application may seem harmless, but together they often have a cumulative effect on usability and therefore should not be ignored.
The Rained-Out tour concentrates on terminating functionality and verifying that the application continues to behave correctly. The two tips that I provide for this tour are actually already mentioned in Chapter 4; however, I want to emphasize them here because they truly are very helpful in detecting bugs.
First, it is important to change the state of the object under test before canceling out of it. Let’s use a form as an example: Do not just open a form and immediately close it. Instead, alter either the form’s or application’s state before closing it. To demonstrate this, here are some actual bugs that I have found in Dynamics AX using this technique:
• I opened a form and then opened a pop-up window that was parented by the form. With the pop-up window still open, I clicked the form’s X button to close it. As a result, the application crashed because the form failed to properly close the pop-up window before closing itself.
• After opening the User Setup form, I left the form open and switched to a different module in the application. Next, I clicked the User Setup form’s Cancel button, which caused the application to crash.
Second, it is imperative to reattempt the same scenario after canceling out of an operation. I used this very technique recently while performing exploratory testing for a new feature planned for the 6.0 release of Dynamics AX and found yet another client-crashing bug. This new feature ensures that creates, updates, and deletes for inner/outer joined data sources occur within a single transaction. While testing updates, I decided to cancel (discard) these changes by clicking the Restore button on the form’s toolbar. As a result of clicking Restore, my changes were discarded and replaced by values in the database table. I then reattempted to update the same record again, and lo and behold Dynamics AX crashed.
Some applications are so large, such as ERP solutions, that it’s overwhelming to even think where to start with the Landmark tour, simply because there are so many features. In some cases, testers may not even be very familiar with other features outside their primary area of responsibility. A tip for combating this issue is to pair up with another person who is an expert in a complementary feature area.
By David Gorena Elizondo
I started working for Microsoft as soon as I graduated from college. (I had done a summer internship for Microsoft a year prior.) I joined as a Software design engineer in Test, and I was part of the first version of Visual Studio Team System in 2005. My job at the time included testing the tools within Visual Studio for unit testing, code coverage, remote testing, and so forth. I am a tester who tests testing tools!
Throughout my four years at Microsoft, I became involved with the different testing approaches and methodologies used inside and outside the company, from test automation to script-based testing, E2E testing, and exploratory testing. I have experimented with a lot of different test techniques. It is in learning and performing exploratory testing that I found my testing passion. I’ve now been using exploratory tours for over a year. Organizing my thinking around the tourist metaphor has substantially increased the number of fixable bugs that I find in features that I’m responsible for testing.
Although all the tours have been valuable to me at one time or another, I’ve seen through experience that certain tours work best under specific circumstances. I will try to share some of my thoughts on when to use them, and the kinds of bugs that I’ve found while testing the test case management solution that I’ve been working on for the last year.
Note that all the bugs I describe here have been fixed and will be unable to plague any users of our test case management system!
I used the exploratory tours with the test case management system that we’ve been developing for the past year and a half on my team. Before I describe how I used the tours, I will describe our product because its features and design affect the way I chose and executed the tours.
The test case management client works hand in hand with a server, from which it basically pulls what we call “work items”; things such as test cases, bugs, and so forth to display for a user to manipulate. Without the server, the client can basically do nothing. Just by knowing this piece of information, you can easily tell that tours such as the Rained-Out tour and the Saboteur yielded a lot of bugs. Imagine canceling a server operation halfway through its completion, or just getting rid of the server itself. Then if you think about it a bit deeper, you realize that a work item (or anything on the server) can be modified at the same time by more than one client. So, the TOGOF tour yielded a good amount of bugs, as well. Updates are happening all the time throughout the application, so the FedEx tour is also one to target.
Complete books on the tours presented in this book could be written. Until then, here are some of my bug-finding experiences.
When you’re working with a client application that works closely with a server, you can assume that any unexpected server activity can cause weird things to happen (both on the server and on the client). Interrupted requests to a server is tricky business, as is refreshing client-side data. Think about it: If you open a page that starts loading information from a server, and you immediately click Refresh, one action is canceled, and a new one starts immediately. The software on both sides of such a transaction needs to be on its toes. That’s why this tour finds tons of bugs. A few of my favorites are (using the exact titles that were entered into our bug management system) as follows:
• Bug: If we cancel initial connect to a project, we can no longer connect to it manually.
Whenever our test case management system is launched, we remember the previous server (that contains the test case and test data repository) that the user was connected to and initiate an automatic connection to this data store. If users want to connect to a different store, they have to cancel this operation. It turns out that the developers didn’t think through the cancellation scenario well and caused an environment variable to be deleted under certain circumstances; so when I loaded a new version of the test data repository, the connection to the server was lost.
• Bug: When deleting a configuration variable and canceling or confirming, you get prompted again.
This would have been a nasty bug to ship with. It happened whenever I tried to delete an existing test repository variable and then canceled the request at a certain UI prompt. Because the Rained-Out tour makes a tester conscious of recognizing complex and timing-sensitive actions, it led me to naturally think of this test case.
The Rained-Out tour makes us think beyond explicit actions. There are times when a feature or a product has to implicitly cancel an action that has already started (often for performance reasons). This was exactly the case for this bug.
• Bug: Plan Contents: Moving between suites does not cancel the loading of tests.
It turns out that whenever we chose a test repository, the application would start loading the test cases that were associated with it. The tour led me to think that there had to be an implicit cancel action whenever I quickly chose a different repository, and if not, there would be performance issues. It turned out to be true; we were not canceling the action properly, and performance was really bad.
The Rained-Out tour makes it clear to the tourist: Cancel every action you can, and cancel it many times and under many different circumstances. It is raining, so scream and yell for everyone to cancel their plans! This strategy led me to bugs like the next one.
• Bug: Time to refresh starts growing exponentially as we try it several times.
Knowing that a refresh action will basically cancel any in-progress activity, I decided to click that Refresh button many times (quickly), and this actually caused horrible performance issues in the product.
The strategy used to find this next bug was exactly the same one mentioned in the previous one.
• Bug: Stress-refreshing test settings manager crashes Camano.
However, the result was much better for a tester’s perspective: a crash! Clicking the Refresh button like a maniac actually caused the application to die.
The Saboteur forces you to think though the application’s use of resources so that you can vary the amount of resources available, and thus potentially find scenarios that will cause it to fail, as discussed here:
• Bug: Camano crashes when trying to view a test configuration when there is no TFS connection.
TFS is the server used to store test cases and test data. The Saboteur caused me to think through many scenarios where availability of TFS was crucial and where good error-handling routines needed to be in place. Making the server unavailable at the right places sometimes found serious crashing bugs, and the resulting fixes have made our product robust with respect to server connection problems.
In this next bug, the strategy was the same one: looking for a resource that the application uses.
• Bug: Camano crashes at startup time if Camano.config becomes corrupt. Camano continually fails until the config file is corrected.
The resource in this case was a configuration file that the application uses to persist data between sessions. The Saboteur requires that all such state-bearing files be tampered with to see whether the application is robust enough to handle it when those persistent resources are corrupt or unavailable. Playing around with these persistent files not only found some high-severity bugs, it was also fun and had me anticipating when and where the crash would occur. Given that no one else on our team had thought to try these scenarios made me happy to have the Saboteur in my tool kit.
This next bug shows the exact same strategy as the previous one but with a new twist.
• Bug: Camano crashes when config file is really large.
Finding this particular bug consisted of playing around with the configuration file and creating variants of it and modifying its properties. The Saboteur suggests making it read-only, deleting it, changing its type, and so forth. But the property that worked for me was its size. When I created a very large config file, the application was simply not able to cope with it.
Our test case management solution processes a great deal of data that flows freely between the client and the server: bugs, test cases, test plans, test plan content, and so on. All this information has to be refreshed appropriately to remain in sync, and this is not easy for an application where multiple activities work with the same artifacts simultaneously. The FedEx tour is tailor-made to help a tester think through such scenarios. The following bugs show some defects that the FedEx tour found.
• Bug: Test plan not refreshing automatically after coming back from a work item.
The test plan contents show the actual test cases in the test plan. The strategies that the tour led me to were actually modifying properties on test cases (and the test plan itself) and ensuring that all these were refreshed appropriately. It turns out that if we modified the name of a test case, and came back to view the test plan, you had to manually refresh the activity for the test case to show the updated name.
We found a number of bugs similar to this next one.
• Bug: When a test plan is selected in the TAC, and we modify one of its artifacts, Camano crashes.
In this particular scenario, modifying a property (such as configuration name) of a test plan, while showing that same property on another activity, would crash the application because it would not be able to cope with the property change. If you think about it, this bug was found using the exact same strategy as the previous one: modifying properties and artifacts, and making sure they were refreshed correctly somewhere else.
This next one was an interesting bug.
• Bug: Camano will crash forever and ever if we have a test plan that uses a build that has been deleted.
Our test plans can be bound to builds, meaning that we can link a test plan to a particular build. However, if we deleted the build that a test plan was bound to, the application would crash every time we opened that particular test plan. Here the FedEx tour is helping us identify those types of data dependencies and guiding us through thinking about such associations between data elements in a methodical manner.
The TOGOF tour will find bugs in applications that can be used by multiple simultaneous users. We found the following bug while having multiple users active within the application at the same time.
• Bug: Test Configuration Manager: Camano crashes when you “Assign to new test plans” if the configuration is not up-to-date.
Test configurations have a Boolean property called “Assign to new test plans” (which can be toggled on or off). It turns out that if user A and user B were looking at the same exact copy of any given configuration (with say, the Boolean property set to true), and user A changed the property to false (and saved it), whenever user B tried to make any change to the configuration and saved it, his application would just crash. This shows a bug that would have been very difficult to catch if only one user was testing the application. The TOGOF tour’s strategy is very clear in these kinds of scenarios: Test different instances of the application at the same time to find bugs that will be difficult to find otherwise.
By Shawn Brown
In 2000, Microsoft shipped a product that could fit on a device to be carried around and perform many of the same functions that a full-size PC could perform. This device was called the Pocket PC and started a series of releases of Windows Mobile. Throughout the releases of Windows Mobile, more and more functionality was added, and therefore the ecosystem to test became more and more complex: from the first, nonconnected PDA-style device, to the multiconnected GSM/CDMA, Bluetooth, and WiFi devices that could stay connected and provide up-to-date information without the user even having to make a request.
During this evolution, the testing of these devices also had to evolve. Constraints such as memory, battery life, CPU speed, and bandwidth all had to be considered when developing and testing for this platform. In addition, being the first multithreaded handheld device allowed for more functionality and more applications working together to give the user more “smart” behavior, and hence the name Smartphone came into the picture. Now, take that environment and add an additional “unknown” variable into the picture called ISVs (independent software vendors). ISVs may use an SDK (software development kit) to create applications on this platform to expand its capabilities and to a buck or two for themselves through revenue, and may thus cause a new testing challenge. These creative and smart ISVs may not abide by certain development practices that internal-to-Microsoft developers are trained to do, or may want to push the boundaries of what the platform can do, and therefore they may potentially cause unexpected issues when deploying their applications to the devices. Some global measures can be put into place, but as a tester on a Mobile platform with a fairly extensive SDK, we cannot ignore the potential of one of these add-ons to affect the overall health of the platform with respect to all the categories of test. Given the fluidity and the overall challenges of testing Windows Mobile, this is a great product to hone testing skills on.
During my career with Windows Mobile, I have owned the testing of Connection Manager, certain Office applications in the earlier releases, and one of my favorites, the phone functionality.
I catch myself hunting for areas of the product that go unnoticed for lengths of time and capitalize on their neglect. When I find a flaw in a product, it creates a sense of accomplishment and job satisfaction. As my career in test evolved, I took notice that in addition to being able to discover new and creative ways to break my product, I was also taking a closer look at how to prevent bugs from getting into the product and was paying more attention to the end-to-end solution of the system. Bug prevention starts with testing the initial design. Roughly 10 percent of all of the bugs found over the years are spec issues. Looking back, this 10 percent, if they had made it into the product, could have caused more bugs as well as more time to complete the project. As you can understand, finding these issues early helps the product and the schedule.
My philosophy to testing is quite basic. Find the weaknesses of the product before anyone else does, and ensure the product’s strengths are highly polished. This approach requires a continual watch on the product and the environment around it that can affect the performance or functionality. How can you look at the environment you are positioning yourself to test in a way that encompasses everyone who is going to interact with it? Then, how do you prioritize your attack?
Independent of the test problem, I begin by defining the problem statement. What is the desired end goal? I determine the variables. I formulate an approach (keeping it simple). After a simple solution is defined, I determine the reusability and agility of the solution. Where are the solution’s trade-offs and weak points? I have used this approach in developing and testing hardware as well as software.
One example of my use of this methodology is when I was developing a shielding technique to enable electronic components to be used in the bore of an MRI. This violent magnetic environment was never meant to have metallic objects inside (because of safety and because of the potential to disrupt the sensitive readings). Knowing the problem statement was “create a method for housing electronics in the bore of an active MRI unit without disrupting the readings more than 5%,” I started by gaining a better understanding of the environment: the strength of the magnetic field, the frequency of the gradient shifting when the unit is in active mode, the method of the readings, and the requirements around what the electronics inside the bore were expected to do. This information narrowed the solution to two main options:
- Create a shield to prevent the magnetic from penetrating into the housing of the electronics.
or
- Create a shield that prevented the RF from escaping from the housing.
After researching the known methods for both, one being a ferrous metal that would prevent the magnetic fields from entering the box, and the other, a highly permeable metal to shield the RF from escaping from the box, I realized that traditional shielding methods would not fit this goal. (Because the RF that would be generated by either material upon placing them inside the bore during active mode would most likely degrade the reading beyond the 5 percent margin.) In addition, the ferrous material could cause more of a hazard to anyone in the room, given the highly concentrated and erratic magnetic field being produced could cause any ferrous metal to be hurled through space at a deathly rate. Therefore, one more option was removed from the equation. Given the material for shielding could not be a solid plate (because of the RF induced by the environment), it had to be unconnected loops. After I calculated the penetration of the RF, the end results were to create a <1cm grid of nonconnecting loops of copper, which would prevent the RF from escaping from the box and would mostly prevent the RF from entering or being generated by the material itself. My approach led to the shielding technique being a success, and I continue to use this process when testing software.
Testing is a continually growing domain, much like any other engineering discipline. To keep increasing your product’s quality, expose how you test to your developers/designers. The more they know about how you are going to test it, the more they will try to create a design that will account for the methods you will use to exploit the gaps in their design. Testing can be difficult, because even though you want to maintain the edge on being able to break any product, you also want to help your developers write better code. Therefore, you must always be one step ahead of your developers. As they learn how you test and you learn how they approach solving algorithm problems, your ability to break their code becomes more and more difficult, Success! However, it is still your job to find the bugs in the product.
While performing exploratory testing on a previous release of Windows Mobile, I caught a race condition with “smart dial” functionality. In this instance, it was known that a side process was being used to do a background search on the search criteria to aid with performance. The larger the data set, the longer it takes this process to complete. This turned out to be an excellent opportunity to use the Rained-Out tour. After loading more than 4,000 contacts onto the device under test, I entered a search string that was expected to end in no results. As the side process was churning away in the background, I changed the search string by deleting one character. The expected result was to still have no matches, but the side process was not done with checking the entire data set, and therefore the next filter, which is designed to check the results of the initial side process, was passed the data that was not checked yet. This resulted in an incorrect IF statement, and data that did not match the search clause was displayed incorrectly. If this bug had not been caught early and through exploratory testing, it could have caused more downstream design challenges.
Another example of using the Rained-Out tour is in an issue found and then fixed in the Bluetooth bonding wizard in Windows Mobile. After creating bonds with a few headsets and peripherals, I proceeded to use the Rained-Out tour during the connection-request phase. When the listed peripherals were all disconnected, there was an option to connect one of them to the phone. I selected one of the BT peripherals and selected the Connect option. I then noticed a time delay between when the connection request was made and when the connection or the failure dialog was actually shown. During this time, I tried all the available menu items. They all functioned properly during this state. It wasn’t until I moved focus away from the existing item, then back to the item with the current connection request, that I noticed that the connection option became available again. So as a tester using the Rained-Out tour, I selected that option. Doing so keyed up another connection request to the same peripheral. I performed the same steps a few more times in quick succession before the initial connection request completed, and voilà, a sequence of failure dialogs popped up. Ultimately, the code was able to handle this situation, but there was no need to be able to continue to make multiple connection requests to the same peripheral, and therefore the bug was fixed.
A contact list is linked to a number of other functionalities (call history, text messaging, speed dial, and so on). Knowing this, I used a Saboteur and created a list of contacts and some speed-dial entries. Now for the unexpected situation to be triggered: A sync error was simulated where the contacts on the device were all removed from one of the linking databases on the device, but the device still thought they were synced (because they still existed on the device). Therefore, syncing was seemingly successful, but the database that housed the links to the speed dial did not exist anymore, and therefore a blank speed dial was shown.
Here is an example of when I used the Supermodel tour on a Windows Mobile device and explored UI centering and anchor points using different resolutions. After booting an image with a resolution that I knew was not used a lot, I navigated around the device performing simple user tasks (for example, creating contacts, checking email, and checking calendar events). During the calendar event navigation, I noticed that when I selected to navigate to a specific date, the calendar week view centered itself onscreen. A number of views could be used, and so I inspected each view. Not until I got to the month view did I see a centering issue. After I changed which month to view via the month picker, the repainting of the month that was selected was centered in the middle of the screen, instead of being top justified like every other view. This incorrect centering occurred because of a missing flag in this one view.
A good time to use the Saboteur is when testing an application that utilizes data connectivity. Data connectivity is a fickle beast and can come and go when you least expect it. Using this tour, I was able to find a problem with a device-side client who requires a connection to provide its functionality. Not only was this application coupled to connectivity, it also had an impact on other locations where the user’s entity can be used. I’m speaking of Instant Messenger, of course. I successfully configured this application and signed in. Now, knowing my device is a 2.5G device, which means it can handle only one data pipe at a time, I then called the device from another phone, thus terminating the active connection, which, on Windows Mobile on GSM, puts the device connection into a suspended state. Then I signed in to the same account on my desktop, attempting to fully sever the connection to that service on the device. Voilà, the device-side client never got the notification that it was signed in to another location but it also appeared to be signed out. However, in the menu options, the option to Sign Out was still available (because it never was told it was signed out).
Another way to be a saboteur is to turn the device’s flight mode on, which turns off all radios (for when the user goes on an airplane but still wants to use the device). Many times, applications do not listen to radio on/off notifications, and this can cause a limbo state to occur. Sure enough, signing on to IM on the device and then turning off the cellular radio, which in turn removes all GSM connectivity, was not detected by this application, and the application is in a perpetual “connected” state, which is quite confusing to the user.
An example of a bug found by using the Supermodel tour coupled with focus specifically on usability and intuitiveness of designs is as follows. While traversing through a connected mapping application on Windows Mobile, I decided to see how easy it was to get directions from my current location to another location, such as a restaurant. I launched the application, the device found my current location, and then I decided to select the option to get Directions from A to B from the menu. In this UI, there was no intuitive way to select my current location as my starting point. Given that I did not know the address where I currently was, I just then decided to enter the point B location and hit Go. An error appeared stating it needed to know where I was starting from. Interesting, because upon entering the application, it made it a point to tell me where I was. This lack of completeness can cause much pain when using the application, and a simple addition to prepopulate this field could be the difference between someone using this application all of the time versus it getting slammed in a review on the Web.
By Bola Agbonile
I am a graduate of the University of Lagos, Nigeria, where I attained an MBA (1996) and a Bachelor’s degree in Electrical Engineering (1990). I presently work on the Windows Experience (WEX) platform with the Windows Media Player team as a Software Development Engineer in Test (SDET), a role that I continue to relish daily.
As an SDET, my role is primarily to work with others on my team to ensure that a high-quality product is what we present to our customers. To achieve this, one of my key roles as an SDET is to validate that the finished product satisfies what our target-market customers require and adheres to the written specifications as outlined by the Program Manager. Another one of my primary roles is to ensure that the product can withstand rigorous tests thrown at it.
In the Windows Experience division, on the Windows Media Experience (WMEX) team, I have worked with others at enhancing Windows Media Player (WMP) from previous versions, with the sole aim being to satisfy our target market by providing a solid, robust, functional, media player. For instance, it was with WMP 10 that we first introduced being able to synchronize to devices, and with WMP 11 we took it one step further by adding greater support for Media Transfer Protocol (MTP) devices, including the ability to have an automatic sync partnership and allowing for synchronization from the device. WMP allows for sync, burn, rip, and playback of numerous file types, including pictures and DVDs. WMP 12 introduces a lightweight player mode for quick, easy, and clutter-free playback with practically no UI chrome.
WMP is an application that is UI-centric. As such, the tours relevant to this kind of application are the ones I use. To be specific, WMP’s input sources are via text boxes, check boxes, option buttons, and “shiny discs” (CDs, DVDs, and CD-R[W]s), while its output is as audio, video, and dialogs displayed to the user.
Following are the tours that I use for testing WMP 12, with examples of things that I have encountered along the way that I consider to be interesting.
Garbage collectors are systematic, house to house, driveway to driveway. Some testers might, as Chapter 4 suggests, be systematic by testing features that are close together in an application. I apply it a bit differently and arrange features in buckets according to their similarity. For WMP, the first categorization bucket could be “All UI Objects.” The next bucket will be “Dialogs,” followed by “Text Boxes,” “Boundaries,” and so on. Then, the garbage collector can begin her systematic collection.
WMP buckets look something like this:
- WMP’s Player mode
a. Transport control
1. Shuffle
2. Repeat
3. Stop
4. Back
5. Play
6. Next
7. Mute
8. Volume
b. Buttons
1. Switch to Library mode
2. Switch to Full Screen mode
3. Close WMP
4. Minimize WMP
5. Maximize WMP
c. Seek bar
d. Title bar
e. Right-click context menu
1. Spelling of labels
2. Persistence of user’s choice
3. Navigation using keyboard
f. Hotkey functionality
1. Alt+Enter
2. Ctrl+P
3. Ctrl+H
4. Ctrl+T
5. Ctrl+Shift+C
g. Dialogs
1. Options
Tabs
Options buttons
Check boxes
Text boxes
Command buttons
Next button
Previous button
Hyperlinks
Option buttons
Check boxes
Labels
Hover tooltips
Sliders
Mouse pointer
Drop lists
3. Default language settings
Drop list
Arrow-key navigation
Command buttons
h. List pane
i. Center pane
1. Play all music shuffled
2. Play again
3. Continue playlist
4. Go to Library
5. Play previous playlist
j. External links
1. Windows Help and Support
2. Download visualizations
3. Info Center view
One of the benefits of this tour is that with the categorization of the application, the tester can systematically walk through and identify features that might otherwise be overlooked. For example, while taking this tour and looking WMP over, I came across a bug that I have now logged. The bug is that in the center pane, “Play previous playlist” does not actually start playback unlike “Play all music shuffled” and “Play again.” A user would expect consistent behavior, and if I had not applied the Garbage Collector’s tour, I would have missed this difference.
The Garbage Collector’s tour made me take a long hard look at WMP and systematically categorize all the buttons available in the center pane (for instance) and then test each item’s functionality for consistency when compared with other items in a similar bucket.
This tour should be on your list when success is to be measured by the quantity of bugs found. For example, if there is to be a bug bash, and there are no restrictions on which feature gets tested, it is beneficial to be one of the first people on the ground when the whistle blows. Examples of supermodel bugs that I classify as “low-hanging fruit” are typographical errors. Early on in the product cycle is when this category of bugs is more common. To effectively spot text bugs, one should read each word and then count to two before reading the next word. In my opinion, this is the secret to spotting typos and grammatical errors.
Figure 6.3 is an example of a bug that I found on a WMP dialog. Take a look at the dialog and see whether you can spot the typo in it.
Spot the bug? It is in the second paragraph. “Do you want to allow the Web page full to access your....” This should be, “Do you want to allow the Web page full access to your....” It is easy to read too fast and correct the sentence in your head while not seeing the error in front of you.
When running manual tests, it pays to keep asking “What if?” For instance, when looking at the Player versus Library mode of WMP, one could ask this: “What if the Library mode is busy and then WMP is expected to be in the Player mode?”
Having asked this question, one would then investigate possible answers by performing a task that is exclusively for the Library mode (such as ripping an audio CD) and then performing a task that is exclusively for the Player mode (such as DVD playback). The whole purpose of this is to see whether this scenario has been handled or whether one would get the undesirable situation of WMP running in both modes (see Figure 6.4).
This is one of my favorite tours because it makes me feel like a real detective. The Boundary tour involves conducting tests close to the upper and lower boundaries, all the while looking for a breaking point. It’s a specific case of the Intellectual tour that suggests that we ask the software hard questions.
Classic examples include the following:
• Filling a text box with its maximum number of characters or null
• Creating a very deep folder hierarchy, and then placing a media file in the last folder and attempting to play back the media file in WMP
• Clicking a button very close to its outer edges to see whether the click command will still be recognized
For example, in WMP, after I attempted to type characters in a text box meant for only numbers, the mitigation taken was to ensure that a user can type in only numbers (see Figure 6.5).
Conducting more tests on a different tab unearthed a dialog with bad indentation (see Figure 6.6).
The bugs found while on the Boundary subtour are varied in nature, including buffer overruns, bad handling of data, and UI formatting errors.
By Geoff Staneff
In 2004, I came to Microsoft with zero programming experience, but a Ph.D. in Materials Science from the California Institute of Technology. I spent the first nine months taking computer science courses and test training in the mornings while working on the Windows event log in the afternoons. By the end of the first year, I owned 50k lines of native test code, filed 450 bugs, and had an 80 percent fixed rate. Since moving to Visual Studio, my focus has changed tremendously. Our work, both the product we test and the test code we write, contains predominantly managed code, and most of our regular testing is manual or semi-automated. Although the particulars of the day-to-day work are vastly different, the act of testing is the same. I keep a lab book at my desk to track identified and reproduced defects and to note interesting areas of investigation that bear future attention.
Our development and test teams work closely during each sprint. It is standard practice for the test team to build new bits between check-ins from our developers and participate in code reviews for the same. As such, we have some uncommon advantages in knowing where certain kinds of defects are likely to be found. Instead of jumping right into these known risk areas, I take a more methodical approach ingrained by my years of studying experimental science. The first thing I do with a new application (or version) is take a brief Parking Lot tour to get the lay of the land. As I learn how the various parts of the application are supposed to work together, I take notes on which areas deserve in-depth attention and follow-on tours. The next wave of tours focuses on individual features, typically with an imposed bias such as accessibility, expose all error dialogs, or force all default values (permutations pulled straight out of How to Break Software and summarized in Chapter 3, “Exploratory Testing in the Small,” of this book).
Between the Parking Lot tour and the Breaking Software tour, I generally pick most of the “low-hanging” fruit in terms of obvious bugs.
My final wave of tours is a much more targeted endeavor. After a couple days observing how the product under test responds, it is time to go for more abusive tours that challenge the assumptions of implementation. The ones I end up using the most are the Back Alley tour and the Obsessive-Compulsive tour, built on previously observed reactions in the product (observations or behavior that were technically correct, but perhaps incomplete or narrow) or conversations with the developer about his implementation.
Over the course of two such tight sprints, I identified about 75 defects in each, of which all but 5 were fixed before the end of the week in which they were identified. We were working in what we call “low-overhead mode,” such that if development could address the issue before the end of the week, test would not formally log the bug. This arrangement was great from both sides of the dev-test relationship; product quality was quickly improved, and no one had to deal with bug-reporting and -documenting overhead. Had the turnaround on bugs been days rather than hours, this arrangement would not have been appropriate, because details of the observed bug could have been lost.
Keeping the pace brisk was facilitated by a cut time each day when a new build would be produced and tested. This had a dual impact. First, development would communicate the fixes we should expect prior to the cut time each day, focusing the testing effort to newly implemented or repaired features. Second, the regular testing start time provided regular feedback to development about the state of their progress. Developers were eager to get their fixes into the next day’s testing effort, causing them to work with test to ensure they’d make the test deadline each day, which helped keep the features to spec with limited creep. We maintained a very high rate of code churn and a tight loop of checking in and testing, which allowed us to stay on top of defects and find the important ones early. Testing during these sprints lent itself to exploratory tours, as the testing cycle was iterative and short. Returning to a general tour such as the Money tour or the Guidebook tour every few days helped ensure that we didn’t let any new work slip through undetected. I do not recall any new bugs in those features over the last four months, despite regular use by our team and others.
Reviewing the bugs identified over one of these testing efforts shows a breakdown by detecting tour:
• 9% Taxicab tour (keyboard, mouse, and so on)
• 9% Garbage Collector’s tour (checking resources after releasing/removing them)
• 15% Back Alley tour (attempting known bad actions, such as closing a dialog twice)
• 18% Obsessive-Compulsive tour
• 19% Landmark tour
• 30% Supermodel tour
While most of the Supermodel tour defects were not recall class, nearly all the Back Alley and Garbage Collector’s tours would have forced a recall or patch on a released product. Although the Supermodel tours didn’t directly reveal the more severe issues, they did identify places where more-focused tours could follow up and deal real damage. One such example was the identification of keyboard acceleration for the Cancel button in a wizard. This observation identified a bug immediately. Wizards do not typically accelerate the Cancel button with a lowercase L; instead, they usually reserve the Esc key for that, without any visual indicator. A follow-up Back Alley tour exploited this keyboard acceleration to cancel the wizard several times. This exposed an unhandled exception and a UI timing issue where a modal dialog was hidden behind the UI element waiting on it. Of course, canceling the same wizard instance more than once should also have not been possible.
Whenever I revisit a feature we’ve previously tested, I run another Parking Lot tour. Just recently, using this method I observed two defects in functionality that had been verified not two weeks earlier. Even when I wasn’t planning to test this particular part of the feature, just by revisiting the bounds of the code under test, a broken link and an unhandled exception presented themselves for observation. Following a strict script and just getting in and getting your work done as efficiently as possible would have missed these opportunities, leading to increased mean time to detection for new defects in the system.
The Parking Lot tour was born of countless family vacations, where distances on the map didn’t look nearly as imposing as they did after driving from place to place. Such ambitious plans often saw the family arrive at a venue after closing time, leaving nothing to do but tour the parking lot and try to make the next stop before it too closed. I used a Parking Lot tour to find a usability bug and an app crash during my interview for my current team:
• Primary objective: Identify the entry points for all features and points of interest within the scope of testing.
• Secondary objective: Identify areas where specific tours will be helpful.
• Targets of opportunity: Enumerate any show-stopping bugs on the first pass.
In a way, the Parking Lot tour is like a mixture of a Landmark tour and a Supermodel tour. The first pass isn’t going to look too deep and is really more interested in how the code under test presents itself than anything the code under test does.
By Geoff Staneff
When testers express an interest in exploratory testing, this can often be interpreted by their managers as a rejection of rigor or planning. Through the use of the touring metaphor, managers can gain repeatability and an understanding of what has been tested, while testers can maintain their autonomy and exploratory imitative. This section outlines some of the strategies and techniques that will prove useful when managing a tour-driven test process.
Two concerns frequently arise when discussing exploratory testing from the standpoint of the person who isn’t using it right now in his own work. First, there are concerns about what will be tested. How can we know what we’ve covered and what we’ve not covered when the testers decide where and how to test while they are sitting with the application under test? Second, there are concerns about the transferability of that knowledge. What happens when the exploratory tester becomes unavailable for further testing efforts on this feature or product? Both of these concerns can be addressed through the utilization of testing tours.
What is a testing tour but a description of what and how to test? By taking a well-defined tour, for instance the Supermodel tour, and pointing your defect observer (the tester) at a particular feature or product, you can know that a test pass has been scheduled to review said feature or product to survey the state of fit and finish class defects. Although the tester may, under his or her own initiative, find other kinds of defects, the entire testing act has been biased to detect those defects that make the product look bad.
Where heavily scripted testing may specify precisely which defects to confirm or deny the existence of in a given product, the tour specifies a class of behavior or defect and leaves it to the tester to determine the best route to confirming or denying the existence. This leaves management free to set the testing strategy, without impairing the tester’s ability to choose tactics suitable to the feature or session of exploration.
The second question is perhaps a more difficult question for test as a discipline and not just exploratory testing. As testers gain experience, they improve their understanding of software systems, how they are typically constructed, and how they typically fail. This has the advantage of making experienced testers more efficient at finding typical software defects, or even defects typical of a given piece of software through the various stages of development, but this increased value makes the temporary or permanent loss all the more damaging to the overall test process. Testers will become unavailable throughout the course of development: They can take time off, be transferred to another project or team, change their role within the organization, leave the company entirely, or any number of other things may arise that conspire to deprive your test effort of its most experienced testers. This is where touring really shines, because it defines what and how, such that any rational agent should be able to perform a similar tour and detect similar defects. Tours will not help testers detect subtle changes in typesetting, but tours can inform a tester that they ought to be concentrating on that sort of defect as they traverse the features they’ve been assigned.
A successful tour is one that intentionally reveals a particular class of defect, providing enough detail to bias the detection, but not so much as to narrow the focus too tightly either in feature scope or defect class. As such, it becomes even more important to record side trips and ancillary tours that were spawned by the scheduled tour. It should not be uncommon for your testers to pass up bug-finding opportunities but note them for further study when they can dedicate their focus to this unplanned area of interest. Touring, therefore, instills some discipline on the testing process. No matter if you choose to take the side trip now and resume the tour later, or mark the POI on your map and return after your tour, by making the distinction between on- and off-tour, one can come along later and understand which combination of tours should reveal all the defects observed. It is at this point when a decision can be made to either expand the set of regularly scheduled tours, or to run with the normal set and track whether these side-trip opportunities present themselves to other testers in the future.
Expanding the set may be helpful if you have a wide product and need to split work between new resources. By expanding the set, one is withholding some of the decision-making process from those doing the testing, but this can be critically important to avoid overlap and redundancy in a testing effort.
Tracking the recurrence of side trips in the future makes sense if you will have several opportunities to test different versions of the same application or same kind of application. In the event that a specific side trip does not present itself, it can be assigned as a special tour just to ensure that there really were not any such significant defects present in this testing effort. The subsequent sections walk through the use of tools through various points of an application development life cycle.
Before setting foot in a new city, prepared travelers will have acquired some basic information about their destination. It’s good to know such things as what language is spoken there, if they will accept your currency, and whether they treat foreigners kindly, before you find yourself in unfamiliar surroundings with no plan in place to mitigate the challenges you’ll encounter. This might mean taking one of the survey tours (Landmark or Parking Lot) yourself to get an overview of the key features you’ll be expected to report on throughout the development and maintenance of the application. At the end of the cycle, you’ll want to be able to report any show-stopper defects as well as how wide the happy path for users actually is through your application. This means providing tours aimed at covering well the core scenarios and providing opportunistic tours with great breadth across the application to pick up abnormalities in out-of-the-way corners of the application.
Many tours fit into the start or end of a development cycle naturally. A tour capable of detecting improper implementation of controls, such as a Taxicab tour, should come earlier than one focused on fit and finish, such as the Supermodel tour. Although the individual testers needn’t have any privileged information about the data or class structure of the application under test, they are still capable of revealing defects that are sensitive to these structures. It is therefore important to find systematic errors early, such as misuse of a classes or controls, before the application has had a chance to solidify around this peculiar behavior and fixing the defect becomes too risky, or time-consuming, or the repercussions of the change too poorly understood to undertake a fix.
Early-cycle objectives include the following:
• Find design defects early.
• Find misuse of controls.
• Find misuse of UI/usability.
These objectives lend themselves to tours of intent: those explorations that focus on getting something done rather than doing that thing in any particular way. Tours of intent include the Landmark tour and the Taxicab tour. At the start of the cycle, a testing effort will usually attempt to identify big problems.
Late-cycle objectives include the following:
• Ensure public functions function.
• Ensure user data is secure.
• Ensure fit and finish is up to expectation.
• Characterize the width of the functional feature path.
• Confirm previous defects are not observable.
These objectives lend themselves to tours of specifics: those explorations that focus on a particular something in a particular way. Tours of specifics include the Back Alley tour, Garbage Collector’s tour, Supermodel tour, and Rained-Out tour.
Permutations on tours should be intentional and planned ahead of time. Thinking about taking a Landmark tour, but only using the mouse to navigate your application? Make that decision up front.
At the end of the day, one must still plan and martial testing resources effectively from the start to secure the opportunity to succeed for one’s team, regardless of the testing techniques employed. Often, realizing that a delay exists between development and test, test will undertake a testing effort over parts of the application in isolation rather than biting off an incomplete end-to-end scenario. Even when the application is nearly complete, when it is handed over to test as a full unit, it makes sense to begin with the early-cycle tours because they help identify opportunities for more in-depth tours and allocation of testing resources.
While exploration and permutation are important parts of the exploratory testing process, it is important that the tour and the tour guide stay on target for the duration of the tour. Staying on target means sticking within the intent of the tour. For example, if your current tour is scheduled to hit N major features, make sure you cover those N major features. This advice applies not just to the tester, but also to the test manager: Letting the planned tour run is important to identify the various side trips and follow-up tours that will help guide the next iteration of testing. Tours are meant to be repeated, by different testers, with different permutations in focus.
Because each tour is meant to be short, comprising a session of a few hours, there isn’t much to gain from interrupting the tour. There is, however, much to lose from interrupting a tour in the way of knowing what you were looking for during that session. Testers will find bugs in products. This is not a surprise and isn’t anything out of the ordinary. Figuring out how and why they found that bug might be something to talk about. With a guided tour, you have a framework under which another tester may be able to find similar defects. This is special and should be leveraged in subsequent tour assignments. When testers go off-tour, for whatever reason, they may find defects, but we already expect them to find defects on the tour they were experiencing. By leaving the tour, they will have lost the focus and observational bias granted by that tour, which makes scheduling a sensible follow-up tour much more difficult. A tester who chases down a promising side trip of “good bugs” might not get through the balance of the tour, leaving an even larger risk area in a completely unknown state. Without the confidence to come back to an area despite knowing bugs are ready to be found right now, you’ll put yourself in a state of not knowing what you don’t know at the end of the tour.
Because tours impart a bias to the observer (for example, the tester is interested in specific kinds of observations with respect to the features comprising the tour), it provides great information about both the toured portion of the software and the need for additional tours through those parts of the software. Touring testers will report both opportunities for side trips and defects that were strictly “off-tour” but prominent enough to stand out despite the testing focus elsewhere. This provides several opportunities to involve the testers, helping them to take ownership in the entire process. The opportunities for side trips are clear and can lead directly to specific in-depth tours or additional broad tours in this previously missed feature area. Reports of bugs that reside outside the focus of the tour are indicators that a tour similar to the last but with a focus more attuned to the unexpected bug detected should be performed.
Finally, when multiple testers have taken the same tour, some overlap will occur as to the bugs they report. Because the tour is providing a bias to the act of bug detection, the overlap, or lack thereof, should provide some insight into how many of those kinds of bug remain undetected in the product—or if further tours in this area should be scheduled until such time as the reported bugs from different tourists converge. Assigning multiple testers to the same tour is not actually a waste of resources (in contrast to assigning multiple testers to the same scripted test case, which is a waste of resources). Because the tour leaves many of the tactical decisions of test to the individual, variation will exist between testers despite the similarity in the testing they have performed. Although this might run counter to the claim that a tour will improve reproducibility in a test pass across testers and time, so long as the tour reveals a particular kind of defect with great regularity it is not incompatible with the discovery of different bugs (for example, a variety of manifestations will point back to the same root cause of failure).
When the time comes to report on product quality, a tour-based testing effort will be able to report on what works in the product and how well it works. A tour-based testing effort will be able to report on what fraction of the planned work actually works, in addition to how wide the path to each working feature actually is. From a high level, test will be able to report how likely additional bugs will exist in given features, remaining to be discovered. Likewise, they will be able to report how unlikely certain types of defects will occur along the tour paths taken during testing. While the number of bugs detected per hour might not differ from other testing methods, the tour-based effort should be able to prioritize which types of bugs to find first, and thus provide early detection of assumed risks.
Your circumstances may dictate much of your strategy for breaking down work and running a tour-based testing effort. Some test teams may be involved during the feature design stages and provide input on usability and testability of features before they are implemented, or your team might have no direct contact with the developers, only interacting with “complete” projects thrown over the wall in your general direction.
No matter what deadline you are under, you will want to start with a wider or overview tour to help identify where opportunities for in-depth exploration reside and specific tours to schedule. The feedback loop between these first explorations and the second wave is the most critical, as this is where you will begin to identify the overall allocation of testing resource to the product you have in front of you. Therefore, it is important to have great discipline early in sticking to the tour and identifying places to return with subsequent investigations. First, map the space for which you are responsible, and then go about focusing on particular points of interest or hot spots of trouble.
The touring concepts applied at Microsoft have helped software testers to better organize their approach to manual testing and to be more consistent, prescriptive, and purposeful. All the testers involved in these and other case studies have found the metaphor useful and an excellent way to document and communicate test techniques. They are now focused less on individual test cases and more on the higher-level concepts of test design and technique.
1. Pick any bug described in this chapter and name a tour that would find it. Is there a tour, other than the one cited by the author in this chapter, that would also find the bug?
2. Pick any two bugs in this chapter at random and compare and contrast the bugs. Which one is more important in your mind? Base your argument on how you think the bug would impact a user who stumbles across it.
3. The Supermodel tour has been cited as a good tour for UI testing. Can you describe a way in which this tour might be used to test an API or other such software that has little or no visible UI?