“When you have a good script you’re almost in more trouble than when you have a terrible script.”
—Robert Downey, Jr.
As the preceding two chapters have shown, exploratory testing has a great deal of strategy associated with it. It is a good combination of structured thinking and freestyle exploration that can be very powerful for finding bugs and verifying correctness. This chapter shows how the exploratory testing mindset can be combined with more traditional scenario-based and scripted testing. This hybrid technique relaxes much of the rigidity ordinarily associated with scripting and makes good use of the exploratory testing guidance presented in the last two chapters. It also allows teams that are heavily vested in existing scripts to add exploratory testing to their arsenal.
Traditional scenario testing is very likely to be a familiar concept for the reader. Many testers write or follow some sort of script or end-to-end scenario when they perform manual testing. Scenario testing is popular because it lends confidence that the product will reliably perform the scenario for actual users. The more the scenario reflects expected usage, the more such confidence is gained. The added component that exploratory testing lends to this process is to inject variation into the scenario so that a wider swath of the product gets tested. Users cannot be constrained to just execute the software the way we intend, so our testing should expand to cover these additional scenario variants.
Scenario-based exploration will cover cases that simple scenario testing will not and more accurately mimics real users, who often stray from the main scenario: After all, the product allows many possible variations. We should not only expect that they get used, we should test that they will work.
The idea behind scenario-based exploratory testing is to use existing scenarios (we talk about where to get scenarios in this chapter) much as real explorers use a map to guide themselves through a wilderness or other unfamiliar terrain. Scenarios, like maps, are a general guide about what to do during testing, which inputs to select, and which code paths to traverse, but they are not absolutes. Maps may describe the location of your destination but offer multiple ways to get there. Likewise, the exploratory tester is offered alternate routes and even encouraged to consider a wide range of possible paths when executing a scenario. In fact, that’s the exact purpose of this form of exploratory testing: to test the functionality described by the scenario, adding as much variation as possible. Our “map” isn’t intended to identify the shortest route, it’s intended to find many routes. The more we can test, the better; this leads to more confidence that the software will perform the scenario robustly when it is in the hands of users who can and will deviate from our expectations.
There is no formal definition of scenarios that I know of which really helps testers. Some scenarios are like maps, providing only general guidance, and others are more like printed driving directions with step-by-step instructions for every turn and intersection. In general, scenarios are written prose that follow no fixed format but describe how the features and functionality of the software under test work to solve user problems. A scenario can describe inputs, data sources, environment conditions (things such as Registry settings, available memory, file sizes, and so forth) as well as UI elements, outputs, and specific information about how the software under test is supposed to react when it is used.
The scenarios themselves often originate from outside the tester’s domain. They can be gleaned from artifacts inherited from design and development. Requirements documents and specifications typically describe their purpose in the form of scenarios. Marketing departments sometimes work with scripts for product demos; some forms of agile development require the creation of user stories; requirements are often documented with example scenarios of expected usage. In many cases, testers don’t need to write the scenarios as much as gather them. In fact, recordings made (using capture/replay tools, keystroke recorders, and so forth) during testing are also legitimate scenarios, and thus the tours of the previous chapter can be the source of a great number of high-quality scripts and scenarios. Any and all such scenarios can be used as the starting point for exploration.
In general, a useful scenario will do one or more of the following:
• Tell a user story
Scenarios that tell user stories generally document a user’s motivation, goals, and actions when using the software. User stories aren’t often at the detailed level of “the users clicks here,” but more general, as in “the user enters her banking information.” It’s the job of the tester to bring the scenario down to a level of detail appropriate for a test case.
User stories are an excellent starting point for exploratory testing.
Requirements are capabilities that the software possesses, and written requirements are usually plentiful for software projects of any decent size. Scenarios that describe requirements should talk about how the product is used to perform that capability.
• Demonstrate how a feature works
Scenarios that demonstrate features are usually pretty detailed and specific. They would specify which menus are used, which buttons are pressed, and what data is entered at a fine level of detail. These often appear in online help or printed instructions for users.
• Demonstrate an integration scenario
Products that integrate with other applications or share information often have integration or end-to-end (abbreviated e2e) scenarios defined for them. In this case, scenarios document how features work together and how a user would use those integrated features on some real task.
• Describe setup and installation
Instructions that describe initial installation procedures, setup and configuration, account creation or other administrative tasks, optional installation flags, and customization can readily be used as scenarios for exploratory testing. Guidance from user manuals and online help systems are an excellent source of scenarios for setup and installation.
• Describe cautions and things that could go wrong
Documents that describe troubleshooting and maintenance procedures make very good scenarios. Because these are the features a user would exercise in the event something goes wrong, it is important that they work correctly. Artifacts such as threat models or attack trees that describe tampering attempts are also a good source of such “negative” usage scenarios.
Exploratory testers should work hard to ensure they gather as many scenarios as possible from all of these categories. It is then our task to follow the scenarios and inject variation as we see fit. It is how we choose to inject this variation that makes this task exploratory in nature and that is the subject we turn to next.
Testers often use scenarios that describe user intent to test the software. Scenario testing works because it mimics the way a real user would behave, and thus it finds bugs that, if they survived testing, would plague actual users.
But seldom do real users confine themselves to usage of the software as described by the scenario. Users are free to vary from the scenario by adding steps or taking them away, and they do so according to their own schedules and timetables. It is our task to second-guess such variation and ensure they get tested because they represent some of the most likely ways in which the software will be used after it has been released.
Injecting variation into scenarios is what this form of exploratory testing is all about. A single written scenario can be turned into many individual test cases by methodically considering choices in input selection, data usage, and environmental conditions. Two main techniques are used to accomplish this: scenario operators and tours.
Exploratory testing can be combined with scenario testing to help a tester explore minor and even major variations on a specific scenario. Where a scenario describes specific actions for a tester to take, the techniques described next can be used to permute those actions and create deviations from the scenario that will test different states and code paths. Where a scenario describes general activity, these techniques can be used to select among the possible choices and allow a tester to consider alternate paths in a more methodical manner.
We introduce the concept of scenario operators to achieve this goal. Scenario operators are constructs that operate on steps within a scenario to inject variation into the scenario. When we apply a scenario operator to an existing scenario, we get a new scenario that we call a derived scenario. A tester can apply one or more scenario operators to a given scenario and even apply operators to derived scenarios. The amount and number of such operators is, in true exploratory fashion, up to the individual tester and can be performed in advance of testing or, my preference, on-the-fly.
The scenario operators in the following subsections are the ones most testers will find useful.
Adding additional steps to a scenario can make them more diverse and allow them to test more functionality. Inserting one or more steps into a scenario creates more opportunity for the software to fail. Code paths may be executed with different data, and the state of the software will be varied in ways that are different from what the original scenario allowed. The additional steps can be
• Adding more data: When the scenario asks for, say, 10 records to be added to a database, the tester should increase that to 20 or 30 records or even more if it makes sense to do so. If the scenario requires an item to be added to the shopping cart, add that item and then some additional items on top of that. It is useful also to add related data so that if the scenario calls for a new account to be created, we may also add information to that account over and above what the scenario calls for.
The tester should ask herself, “What data is used in this scenario and how would it make sense to increase the amount of data I enter?”
• Using additional inputs: When the scenario calls for a series of inputs to be entered, find more inputs that can be added. If the scenario asks that the tester create a product review for some online shopping site, the tester can choose to add ratings for other customer reviews, too. The idea is to understand what additional features are related to the features in the scenario and add inputs to test those new features as well.
The tester should ask herself, “What other inputs are related to the inputs used in the existing scenario?”
• Visiting a new part of the UI: When the scenario calls for specific screens and dialog boxes to be used, the tester should identify other screens or dialogs and add those to the scenario. If the scenario calls for a tester to pay a bill on a financial services website, the tester could choose to also visit the pages to check account balances before submitting the payment.
The tester should ask herself, “What other parts of the UI are related to the parts used in the existing scenario?”
Eventually, the steps need to loop back into the original scenario. It helps to keep in mind that the idea is to enhance the scenario, not to change it from its fundamental purpose. If the scenario was meant to add records to the database, that should still be its primary purpose, and that goal should not change. What the tester is doing in this scenario operator is adding inputs, data, or variation that makes the scenario longer but does not alter its core purpose.
Redundant and optional steps can also be removed with the idea being to reduce the scenario to its shortest possible length. The derived scenario may then be missing steps that set preconditions for other steps, testing the application’s ability to recognize missing information and dependent functionality.
A tester can apply this scenario operator in an iterative fashion, removing one step at a time. In this case, the scenario actually gets executed against the software under test each time a step is removed until the minimal test case ends the cycle. For example, a scenario that requires a tester to log on to a shopping site, search for items, add them to a shopping cart, enter account info, complete the purchase, and finally log off would be eventually reduced to just logging on and logging off (an interesting and important case to test!) with a single step being removed each time the test case is run.
If there is more than one way to accomplish some specific step in a scenario, this scenario operator is the way to modify the scenario to accomplish that. It’s really a combination of the preceding two operators in that replacement is the same thing as removing and then adding.
The tester must research alternate ways of performing each of the steps or actions in a scenario. For example, instead of searching for an item to purchase, we might simply use its item number to look it up directly. Because the software under test provides both of these as options, we can create a derived scenario to test the alternative. Similarly, we might use keyboard shortcuts instead of the mouse or choose to bypass creating an account and just purchase an item without registering on the site. Testers need to be aware of all the different options and functionality that exists within their application to be truly effective at applying this scenario operator.
Scenarios often contain very specific sequences of actions. This operator modifies such a sequence by repeating steps individually or in groups to create additional variation. By repeating and reordering steps, we are testing new code paths and potentially finding bugs related to data initialization. If one feature initializes a data value that is used by another feature, the order in which the two features is executed matters, and reordering them may cause a failure.
Often, certain actions make sense to repeat. For example, if we are testing a financial services website for the general scenario of log in to an account, check the balance, pay bills, make a deposit, and then log out, we may repeat the “check the balance” action after we pay the bills, and then again after making the deposit. The general scenario is the same, but we have repeated an action that a user is also likely to do. The same can be said of actions such as “view the shopping cart,” which could happen over and over during a scenario for an online shopping site.
Repetition can also occur with multiple actions, so that we pay one bill, check the balance, pay another bill, check the balance, and so forth. The tester’s task is to understand the variability and create repetitive sequences as appropriate.
It is often the case that a scenario will require a connection to some database, data file, or other local or remote data source. The scenario then specifies actions that the tester performs to cause that data to be read, modified, or manipulated in some way. Testers need to be aware of the data sources that the application under test interacts with and be able to offer variations.
Are there backup databases, alternate test databases, real customer databases, and so forth that are accessible to testers? If so, use those when testing the scenarios instead of the default. What if the data source is down or otherwise unavailable? Can we create or simulate that situation so that we can test how the system under test reacts? What if the data source holds ten times as many records? What if it only holds one record?
The idea here is to understand the data sources the application connects to or uses and to make sure that interaction is robust.
As discussed in Chapter 3, “Exploratory Testing in the Small,” testing is necessarily dependent on the environment in which the software resides when we run our test cases. We can run billions of tests successfully when the software is in one environment only to have them all fail when the software is put into a different environment. Therefore, this operator is used to ensure those alternate environments receive testing.
The simple part of this operator is that the scenarios themselves don’t actually change, only the system on which the software is running when the scenario is applied. Unfortunately, understanding which parts of the environment to change, and actually enacting that change, is very difficult. Here are some considerations:
• Substitute the hardware: The easiest part of the environment to vary is the hardware on which the application under test case runs. If we expect our users to have a range of hardware from fast and powerful to antiquated and slow, we need to acquire similar machines for our test lab and ensure that we have beta customers willing to help us with testing and pre-release validation. Of course, this is an excellent use of virtual machines as well.
• Substitute the container: If our application runs inside a so-called container application (like a browser), we need to ensure that our scenarios run in all the major containers we expect our user to have access to. Browsers like Internet Explorer, Firefox, Opera, and Chrome or platforms like Java or .NET or even animation tools like Flash and Silverlight will impact the way our applications run.
• Swap out the version: All the previous containers also have earlier versions that still enjoy market share. How does your app run in the earlier versions of Flash?
• Modify local settings: Does your application use cookies or write files to user machines? Does it use the local Registry? What happens when users modify their browser settings to limit these types of activity? What happens if they change your application’s Registry settings directly (without going through your app)? If you don’t test these things, your users likely will, and their doing so may bring a nasty post-release surprise to your engineering team. It’s better to find out for yourself before the app ships how it will handle these things.
When using any of these operators to create derived scenarios, it is generally the case that we try to stay as true to the original scenario as possible. Using too many operators or using operators in such a way as to make the origin of the derived scenarios unrecognizable is usually not useful. But don’t take my word for it. If you try it and it finds good bugs, then it’s a useful technique! However, such broader based modification of tours is the job of the second technique to inject variation in scenarios: tours. This is the subject we turn to next.
At any point in the execution of a scenario, one can stop and inject variations that will create derived scenarios. The scenario operators described above are one way to do this, and using the tours is another. I like to think of this use of tours as side trips. The idea is simple: A tester reviews the scripts looking for places where decisions can be made or places where it is possible to fork the logic within the script and go in a completely different direction before returning to the main path dictated by the script.
I like to use the analogy of a car tour or even a hike in the woods on foot. It’s often that on such a trip there is some scenic overlook at which one can park the car and take a short walk to some monument or beautiful view before returning to the car and continuing the voyage. That short side trip represents the tour, and the longer car ride is the scenario. This is a useful technique for adding variation to scenarios.
The key difference between scenario operators and tours is that tours end up creating longer side trips, in general, than operators. Operators focus on small, incremental changes and optional steps in a scenario, and tours can actually create derived scenarios that are significantly longer and broader in scope. Just as some side trips can turn into a destination all their own, it may be that the tours overwhelm the original scenario, and this can actually be a very desirable effect. It’s good to always remember that exploratory testing is about variation, and when scenarios are combined with tours, the result can add significant variation. It’s up to the tester to determine whether the variation is useful, and it is often the case that one has to build up some history to determine which tours are most effective for a given application.
Here is a list of tours that are effective as side trips during scenario-based exploratory testing. I suggest rereading the actual tour as presented in Chapter 3 along with the additional description here. After you practice the tours a few times, you should be able to determine how to best follow this advice for your particular situation.
Can any major features not already used in the scenario be easily incorporated into the scenario? If so, modify the scenario to include the use of a new feature or features. Assuming that the original scenario already included some features, this will help test feature interaction in a scenario-driven way. If the scenario was a realistic user scenario, it’s even better because we are mimicking the user including another feature into his existing work habits (as represented by the scenario). There are many users who will learn a feature, master it, and then move on to new features as their familiarity with the application grows. This technique mimics that usage pattern.
Start with a scenario and pick specific feature landmarks out of the scenario. Now randomize the order of the landmarks so that it is different than the original scenario. Run some tests with the new order of landmark features and repeat this process as often as you think is necessary. Obviously, that will depend on how many landmarks you are dealing with; use your own judgment. This combination of the Landmark tour within a structured scenario has been very valuable at Microsoft.
Review the scenario and modify it so that it makes the software work harder. In other words, ask the software hard questions. If the scenario requires the software to open a file, what is the most complicated file you can give it? If the software asks for data, what is the data that will make it work the hardest? Would very long strings do the trick? What about input that breaks formatting rules (for example, Ctrl characters, Esc sequences, and special characters)?
This is an interesting variation on the Money tour. Both tours suggest we inject new features into the scenario, but the Back Alley tour suggest the least likely or least useful features instead. Granted, this variation will find more obscure bugs, but if an application is widely used, there may be no such thing as least likely because every single feature will get used by someone, and all paying customers are important.
This one is straightforward: Repeat every step of the scenario twice. Or three times. Be as obsessive as you like!
Specifically, any step in a scenario that manipulates data is a good one to repeat because it will cause internal data to be manipulated and internal state to be set and then changed. Moving data around the software is always an effective way to test and to find important bugs.
This one is best when a scenario can be automated or even recorded and then played back. Just run the scenario over and over without ever exiting the application under test. If the scenario specifies that the software be shut down, remove that clause and keep the scenario running over and over again. Choose scenarios (or even derived scenarios) that make the software work hard, use memory and the network, and otherwise consume resources that might over time cause problems.
Scenarios are a great start for sabotage. Review the scenario or derived scenario and make a note every time it uses some resource (another computer, the network, file system, or another local resource) that you have access to, and then when you execute the scenario, sabotage that resource when the scenario calls for it to be used.
For example, if a scenario causes data to be transmitted over a network, unplug the network cable (or disconnect it via the OS or turn off the radio switch for wireless connections) just before or while you are executing that particular step of the scenario. Document all such sabotage points and execute as many of them as sensible or prudent.
Document every output you see as you execute scenarios and derived scenarios. You can even score scenarios based on the number of such outputs they force. The more outputs, the higher the score for that scenario. Can you create (or derive) new scenarios that cause outputs that are not in any of the other scenarios? Can you create a super scenario that causes the absolute maximum number of outputs possible? Make a game out of it and let your testers compete to see who can generate the most outputs, and give prizes to the winners.
Run the scenario but don’t look past the interface. Make sure everything is where it is supposed to be, that the interface is sensible, and watch particularly for usability problems. Choose scenarios that manipulate data, and then cause it to be displayed on the UI. Force the data to be displayed and redisplayed as often as possible and look for screen-refresh problems.
I think of this as the Nearest-Neighbor tour, in that instead of exercising the features as described in the script, the testers find the nearest neighboring feature instead.
For example, if a scenario specifies an item on a drop-down menu, choose the item above or below the one specified. Whenever a choice is presented in the scenario, choose not the one suggested but one right next to it (either by proximity on the interface or close in semantic meaning). If the scenario specifies using italics, use boldface; if it wants you to highlight some text, highlight other text instead, always choosing that which is “nearest” in whatever way makes the most sense.
This is the tour that not only makes good use of the cancel button (press it whenever you see it while running the scenario) but also in starting and stopping execution. Review the scenarios for time-consuming tasks such as complicated searches, file transfers, and the like. Start those features, and then cancel them using provided cancel buttons, hitting the Escape key and so forth.
This tour is new for this chapter and didn’t appear earlier when the tourist metaphor was first described. Indeed, it is specific to scenario-based testing. The concept is based on those people who don’t pay for the tour when it begins, but join it in progress by simply melting into the crowd and acting like they’ve been there all the time. They not only crash a tour, but they also may even hop from tour to tour as they encounter other groups (in a museum or some historical building where tours are continuous) of tourists.
We’re going to adopt this process for hopping from scenario to scenario as a way of combining two or more scenarios into a single scenario of mixed purpose. Review your scenarios and find ones that operate on common data, focus on common features, or have steps in common. It is this overlap that will allow you to seamlessly leave one scenario and pick up the remainder of another. Just like the guy who peels himself away from one tour and melts into the crowd of another. He’s able to do it because for some small period of time, the two tour groups are sharing the same space on the museum floor. We’re able to do it as testers because the scenarios both go through the same part of the application. We’ll follow one scenario to that place but then follow the other when we leave it.
Static scenario testing and exploratory testing do not have to be at odds. Scenarios can represent an excellent starting point for exploration, and exploration can add valuable variation to otherwise limited scenarios. A wise tester can combine the two methods for better application coverage and variation of input sequences, code paths, and data usage.
1. Name the two ways to create derived scenarios from existing scripts or scenarios as described in this chapter. Which one do you think would be likely to find the most bugs? Justify your answer.
2. Name and describe at least three software development artifacts from which scenarios can be gathered. Can you come up with a way to create a scenario that is not described in this chapter?
3. When creating derived scenarios, what is the primary difference between using tours and using scenario operators? Which will yield the most variation from the original scenario?
4. Which scenario operator is related to the Obsessive-Compulsive tour? How would one get the same derived scenario from this operator and the OCD tour?
5. Pick a tour from the previous chapter that is not used in this chapter and see whether you can reason about how it might be used effectively in scenario-based exploratory testing.
6. What properties of a scenario would make it a good candidate for the All-Nighter tour? Why?