Sampling from Production

The most common test data acquisition technique is to take it from production. This approach seems both logical and practical: production represents reality, in that it contains the actual situations the software must deal with and it offers both depth and breadth while ostensibly saving the time required to create new data.

There are at least three major drawbacks, however. The test platform seldom replicates production capacity, and so a subset must be extracted. Acquiring this subset is not as easy as taking every Nth record or some flat percentage of the data: the complex interrelationships between files means that the subset must be internally cohesive. For example, the selected transactions must reference valid selected master accounts, and the totals must coincide with balances and histories. Simply identifying these relationships and tracing through all of the files to ensure that the subset makes sense can be a major undertaking in and of itself. Furthermore, it is difficult to know how large a sample is necessary to achieve coverage of all critical states and combinations.

The second major drawback of this approach is that the tests themselves and the extracted data must be constantly modified to work together. Going back to our basic tenet, we must know the input conditions for a valid test, in this case, the data contents. Each fresh extraction resets everything. If a payroll tax test requires an employee whose year-to-date earnings will cross over the FICA limit on the next paycheck, for example, the person performing the test must either find such an employee in the subset, modify one, or add one. If the test is automated, it too must be modified for the new employee number and related information. Searching for an employee who meets all the conditions you are interested in is like searching for a needle in a haystack. Thus, the time savings are illusory because there is limited repeatability: all effort to establish the proper test conditions is lost every time the extract is refreshed.

And finally, this approach obviously cannot be employed for new systems under development, inasmuch as no production data is available.

Starting from Scratch

The other extreme is to start from scratch, in effect reconstructing the test data each time. This approach has the benefit of complete control; the content is always known and can be enhanced or extended over time, preserving prior efforts. Internal cohesion is ensured because the software itself creates and maintains the interrelationships, and changes to file structures or record layouts are automatically incorporated.

However, reconstructing test data is not free from hazards. The most obvious is that, without automation, it is highly impractical for large-scale applications. Less obvious is the fact that some files cannot be created through online interaction: they are system-generated only through interfaces or processing cycles. Thus, it may not be possible to start from a truly clean slate.

A compelling argument also might be made that data created in a vacuum, so to speak, lacks the expanse of production: unique or unusual situations that often arise in the real world may not be contemplated by test designers. Granted, this technique allows for steady and constant expansion of the test data as necessary circumstances are discovered, but it lacks the randomness that makes production so appealing.

Seeding the Data

Seeding test data is a combination of using production files and creating new data with specific conditions. This approach provides a dose of reality tempered by a measure of control.

This was the strategy adopted by a major mutual fund to enable test automation. Without predictable repeatable data there was no practical means of reusing automated tests across releases. Although much of the data, such as funds, customers, and accounts, could be created through the online interface, other data had to be extracted from production. Testing statements and tax reports, for example, required historical transactions that could not be generated except by multiple execution cycles. So, the alternative to acquiring the data from production and performing the necessary maintenance on the tests proved to be less time consuming. Once the data was assembled, it was archived for reuse.

It is still not easy. You must still surmount the cohesion challenge, ensuring that the subset you acquire makes sense, and you must still have an efficient means of creating the additional data needed for test conditions. Furthermore, you must treat the resulting data as the valuable asset that it is, instituting procedures for archiving it safely so that it can be restored and reused.

Although a popular and sensible concept, reuse brings its own issues. For time-sensitive applications, which many if not most are, reusing the same data over and over is not viable unless you can roll the data dates forward or the system date back. For example, an employee who is 64 one month may turn 65 the next, resulting in different tax consequences for pension payouts.

Furthermore, modifications to file structures and record layouts demand data conversions, but this may be seen as an advantage because, it is hoped, the conversions are tested against the test bed before they are performed against production.

Generating Data Based on the Database

Generated test data can obviously be used to create databases with enough information to approximate real-world conditions for testing capacity and performance. If you need to ensure that your database design can support millions of customers or billions of transactions and still deliver acceptable response times, generation may be the only practical means of creating these volumes.

Test data generators begin with the description of the file or database that is to be created. In most cases, the tools can read the database tables directly to determine the fields and their type, length, and format. The user can then add the rules, relationships, and constraints that govern the generation of valid data.

Standard “profiles” are also offered, which can automatically produce billions of names, addresses, cities, states, zip codes, Social Security numbers, test dates, and other common data values such as random values, ranges, and type mixes. User-customizable data types are also available in most products, which can be used for generating unique SIC (standard industrialization classification) business codes, e-mail addresses, and other data types.

A more critical feature, and more difficult to implement, is support for parent/child and other relationships in complex databases. For example, a parent record, such as a customer account master, must be linked with multiple child records, such as different accounts and transactions. This type of functionality is essential for relational database environments in which referential integrity is key.

Some users have found it easier to use a test data generator to create data that is then read by an automated test tool and entered into the application. This is an interesting combination of data generating and seeding. The synergy between test data generation and test automation tools is natural, and in some cases the test data generation capability is being embedded in test execution products.

Databases can contain more than just data, such as stored procedures or derived foreign keys that link other tables or databases. In these cases, it is not feasible to generate data directly and populate the tables. Too often, maintaining database integrity is a project in itself.

And, of course, in the end, volume is its own challenge. More is not necessarily better. Too much data will take too long to generate, will require too much storage, and may create even more issues than not having enough data will.

A Cutting-Edge Test Case Generator Based on Requirements

Because most test efforts require hundreds, if not thousands, of test cases, an extensive development effort is the result when a capture/replay tool uses a scripting language. This is time consuming, as automating an hour of manual testing can require ten hours of coding and debugging. The net result is another development effort within the test cycle, which is not planned, staffed, or budgeted for. For testers who do not have a programming background, there is a steep learning curve to learn how to use these tools.

As the inevitable changes are made to the application, even minor modifications can have an extensive impact on the automated test library. A single change to a widely used function can affect dozens of test cases or more. Not only do the changes have to be mapped to the affected scripts and any necessary modifications made, but the results also have to be tested. Eventually, the maintenance effort takes so much of the test cycle time that testers are forced to revert to manual testing to meet their deadlines. At this point, the tool becomes shelfware.

The focus of future automated testing tools will have a business perspective rather than a programming view. Business analysts will be able to use such tools to test applications from a business perspective without having to write test scripts.

Instead of requiring one to learn a scripting language, or to document their tests so someone else can code them into scripts, the most advanced automated testing tools let one document and automate in one step with no programming effort. Application experts with business knowledge will be able to learn to develop and execute robust automated tests using simple drop-down menus.

As pointed out previously, the focus of test automation to date has been on getting the scripts to work. But where does the data come from? How does the tester know the right test data is being used by the script and how does the tester know there is adequate requirements coverage by the data?

This is a fundamental element that has typically been missing from capture/replay automated tools up to now. What is not being addressed is the quality of the test data input and test scenarios to the scripts. Typically, the automated testing tool scripter is assigned to the testing department. Because this is a very specialized and intensive programming effort, the focus is on getting the script working correctly. It is assumed someone else will provide the test data/test scenarios.

The following is a description of a tool that derives the test data directly from the test objects and requirements. The output of such tools is the input to the automated testing tools. Such a tool bridges the gap between the requirements and testing phases.

Smartware Technology’s SmartTest^™ is an example of such a futuristic tool and has the following major capabilities:

1. ■ Enables manual inputs of parameters and values or imports from Excel

2. ■ Generates pairwise test data that can be input as variable data to automated tools (or manual testing) offered by the following:

   1. –   HP’s Quick Test Pro

   2. –   IBM’s Computer Associate’s CA Verify

   3. –   Compuware’s TestPartner

   4. –   Empirix’s e-Test Suite

   5. –   Segue’s SilkTest

3. ■ Dynamically applies requirements/business rules to the pairwise test data

4. ■ Provides traceability between test cases and the requirements (business rules)

Figure 34.1 shows the SmartTest^™ tree structure in which the test cases are organized and housed. It also shows an Excel-like table format that contains the parameters and associated values. In this example, a parameter is a row in a GUI mortgage application along with the respective values. These parameters are entered either manually or through screen scraping of the target application GUI.

Figure 34.2 shows the SmartTest^™ business rule builder. The rules are entered by pointing at and clicking on predefined parameters and associated values to build a Structured English if–then–else format. A built-in parameter called “Expected Result” enables a rule to self-define the expected result from a test.

Figure 34.3 shows how SmartTest^™ generates test cases and test data with pairwise interactions of the parameters and values. Each row is a test case. The business rules are then dynamically applied to each test row. This adjusts the tests to reflect the requirements (or business rules). Negative and positive test data can be constructed.

Figure 34.4 shows a bidirectional SmartTest^™ report that associates which test cases are associated with each business rule and vice versa. This feature is very useful during maintenance to identify the test cases that should be executed based upon the change on a requirement. (See www.smartwaretechnologies.com for more details.)