G1: Basis Path Testing

Basis path testing is a white-box technique that identifies test cases on the basis of flows or logical paths that can be taken through a program. A basis path is a unique path through the program where no iterations are allowed. Basis paths are atomic-level paths, and all possible paths through the system are linear combinations of them. Basis path testing also produces a cyclomatic metric, which measures the complexity of a source code module by examining the control structures.

Consider the following small program, which reads records from a file and tallies the numerical ranges of a field on each record to illustrate the technique.

G2: Black-Box Testing

Black-box, or functional, testing is one in which test conditions are developed on the basis of the program or system’s functionality; that is, the tester requires information about the input data and observed output, but does not know how the program or system works. Just as one does not have to know how a car works internally to drive it, it is not necessary to know the internal structure of a program to execute it. The technique focuses on testing the program’s functionality against the specification. With black-box testing, the tester views the program as a black-box and is completely unconcerned with the internal structure of the program or system. Some examples in this category include the following: decision tables, equivalence partitioning, range testing, boundary value testing, database integrity testing, cause-effect graphing, orthogonal array testing, array and table testing, exception testing, limit testing, and random testing.

A major advantage of black-box testing is that the tests are geared to what the program or system is supposed to do, and it is natural and understood by everyone. This should be verified with techniques such as structured walkthroughs, inspections, and JADs. A limitation is that exhaustive input testing is not achievable, because this requires that every possible input condition or combination be tested. In addition, because there is no knowledge of the internal structure or logic, there could be errors or deliberate mischief on the part of a programmer that may not be detectable with black-box testing. For example, suppose a disgruntled payroll programmer wanted to insert some job security into a payroll application he is developing. By inserting the following extra code into the application, if the employee were to be terminated, that is, if his employee ID no longer exists in the system, justice would sooner or later prevail.

G3: Bottom-Up Testing

The bottom-up testing technique is an incremental testing approach in which the lowest-level modules or system components are integrated and tested first. Testing then proceeds hierarchically to the top level. A driver, or temporary test program that invokes the test module or system component, is often required. Bottom-up testing starts with the lowest-level modules or system components with the drivers to invoke them. After these components have been tested, the next logical level in the program or system component hierarchy is added and tested driving upward.

Bottom-up testing is common for large complex systems, and it takes a relatively long time to make the system visible. The menus and external user interfaces are tested last, so users cannot have an early review of these interfaces and functions. A potential drawback is that it requires a lot of effort to create drivers, which can add additional errors.

G4: Boundary Value Testing

The boundary-value-testing technique is a black-box technique that focuses on the boundaries of the input and output equivalence classes (see Equivalence Class Partitioning Testing). Errors tend to congregate at the boundaries. Focusing testing in these areas increases the probability of detecting errors.

Boundary value testing is a variation of the equivalence class partitioning technique, which focuses on the bounds of each equivalence class, for example, on, above, and below each class. Rather than select an arbitrary test point within an equivalence class, boundary value analysis selects one or more test cases to challenge each edge. Focus is on the input space (input equivalence classes) and output space (output equivalence classes). It is more difficult to define output equivalence classes and, therefore, boundary value tests.

Boundary value testing can require a large number of test cases to be created because of the large number of input and output variations. It is recommended that at least nine test cases be created for each input variable. The inputs need to be thoroughly understood, and the behavior must be consistent for the equivalence class. One limitation is that it may be very difficult to define the range of the equivalence class if it involves complex calculations. It is, therefore, imperative that the requirements be as detailed as possible. The following are some examples of how to apply the technique.

G5: Branch Coverage Testing

Branch coverage, or decision coverage, is a white-box testing technique in which test cases are written to ensure that every decision has a true and false outcome at least once; for example, each branch is traversed at least once. Branch coverage generally satisfies statement coverage (see G31, “Statement Coverage Testing”), because every statement is on the same subpath from either branch statement.

Consider the following small program, which reads records from a file and tallies the numerical ranges of a field on each record to illustrate the technique.

G6: Branch/Condition Coverage Testing

Branch/condition coverage is a white-box testing technique in which test cases are written to ensure that each decision and the conditions within a decision take on all possible values at least once. It is a stronger logic-coverage technique than decision or condition coverage because it covers all the conditions that may not be tested with decision coverage alone. It also satisfies statement coverage.

One method of creating testing cases using this technique is to build a truth table and write down all conditions and their complements. Then, if they exist, duplicate test cases are eliminated. Consider the following small program, which reads records from a file and tallies the numerical ranges of a field on each record to illustrate the technique.

G7: Cause-Effect Graphing

Cause-effect diagrams (also known as Ishikawa or Fishbone diagrams) are useful tools to analyze the causes of an unsatisfactory condition. They have several advantages. One is that they provide a visual display of the relationship of one cause to another. This has proved to be an effective way to stimulate ideas during the initial search. Another benefit is that they provide a way to keep searching for root causes by asking why, what, where, who, and how. Yet another benefit is that they are graphical representations in which the cause-relationships are easily discernible.

One application of cause-effect graphs was undertaken to understand the inspection process. It discovered that (1) excessive size of materials to be inspected leads to a preparation rate that is too high, (2) a preparation rate that is too high contributes to an excessive rate of inspection, and (3) an excessive rate of inspection causes fewer defects to be found. This analysis using cause-effect graphics provided insights to optimize the inspection process by limiting the size of materials to be inspected and the preparation rate.

Proper preparation for construction of cause-effect diagrams is essential. Visibility is a key requirement. It is advisable to leave a good deal of space between the causes as they are listed, so there can be room for additional notation as the work continues.

Several stages of construction should be expected before a “finished” product is developed. This often consists of enlarging a smaller section of the cause-effect diagram by taking one significant cause and making it the “effect” to be analyzed on another cause-effect diagram.

Cause-effect graphics can also be applied to test case design, particularly function testing. They are used to systematically select a set of test cases that have high probability of detecting program errors. This technique explores the input and combinations of input conditions of a program to develop test cases, but does not examine the internal behavior of the program. For each test case derived, the technique also identifies the expected output. The input and output are determined through the analysis of the requirement specifications (see Section 6, “Modern Software Testing Tools,” which automates the process).

The following is a brief overview of the methodology to convert requirements to test cases using cause-effect diagrams. It is followed by an example of how to apply the methodology.

G8: Condition Coverage

Condition coverage is a white-box testing technique in which test cases are written to ensure that each condition in a decision takes on all possible outcomes at least once. It is not necessary to consider the decision branches with condition coverage using this technique. Condition coverage guarantees that every condition within a decision is covered. However, it does not necessarily traverse the true and false outcomes of each decision.

One method of creating testing cases using this technique is to build a truth table and write down all conditions and their complements. If they exist, duplicate test cases are eliminated.

Consider the following small program, which reads records from a file and tallies the numerical ranges of a field on each record to illustrate the technique.

G9: CRUD Testing

A CRUD matrix, or process/data matrix, is optionally developed during the analysis phase of application development, which links data and process models. It helps ensure that the data and processes are discovered and assessed. It identifies and resolves matrix omissions and conflicts and helps refine the data and process models, as necessary. It maps processes against entities, showing which processes create, read, update, or delete the instances in an entity.

The CRUD matrix in Exhibit G.3 is developed at the analysis level of development before the physical system or GUI (physical screens, menus, etc.) has been designed and developed. As the GUI evolves, a CRUD test matrix can be built, as shown in Exhibit G.3. It is a testing technique that verifies the life cycle of all business objects. In Exhibit G.3, each CRUD cell object is tested. When an object does not have full life-cycle operations, a “—” can be placed in a cell.

A variation of this is to also make unit performance measurements for each operation during system fragment testing.

G10: Database Testing

The following subsections provide a description of how to test databases. An overview of relational database concept is also presented, which will serve as a reference to the tester.

G12: Desk Checking

Desk checking is a human error-detection process in the form of a one-person walkthrough. The typical application is where an individual reads a program, verifies it with a checklist, and manually walks test data through it. It can also be applied to requirements and design as a check on the work. This technique provides an evaluation of the quality of the program after it has been written or after the design has been completed.

G13: Equivalence Partitioning

Equivalence partitioning is a black-box testing technique that partitions the input domain into a set of input classes that can cause multiple behaviors.

From the requirements, each input is divided into partitions. Using this technique, one representative value from each partition is selected and tested. It is assumed that the results predict the results for other values in the partition, which demonstrates the power and economy of this technique.

It is more complicated than a simple range test because a range is divided into a series or one or more ranges because of the different behaviors that can occur. Consider the following application. The income needs to be broken up into three equivalence classes, as the behavior (or tax) varies according to the income value.

An IRS program computes the amount of state income tax on the basis of the income, as is displayed in Exhibits G.36 and G.37. The following are some guidelines for defining equivalence classes.

G14: Exception Testing

With exception testing, all the error messages and exception-handling processes are identified, including the conditions that trigger them. A test case is written for each error condition. A test case/error exception test matrix (Exhibit G.38) can be helpful for documenting the error conditions and exceptions.

G15: Free-Form Testing

Free-form testing, often called error guessing, ad hoc testing, or brainstorming, is a “blue-sky” intuition of where and how errors are likely to occur and is an add-on technique to other testing techniques.

Some testers are naturally adept at this form of testing, which does not use any particular testing technique. It involves intuition and experience to “smell out” defects. There is no particular methodology for applying this technique, but the basic approach is to enumerate a list of potential errors or error-prone situations and write test cases on the basis of the list.

G16: Gray-Box Testing

Black-box testing focuses on the program’s functionality against the specification. White-box testing focuses on the paths of logic. Gray-box testing is a combination of black- and white-box testing. The tester studies the requirements specifications and communicates with the developer to understand the internal structure of the system. The motivation is to clear up ambiguous specifications and “read between the lines” to design implied tests. One example of the use of gray-box testing is when it appears to the tester that a certain functionality seems to be reused throughout an application. If the tester communicates with the developers and understands the internal design and architecture, a lot of tests will be eliminated, because it might be possible to test the functionality only once. Another example is when the syntax of a command consists of 7 possible parameters that can be entered in any order as follows:

Command parm1, parm2, parm3, parm4, parm5, parm6, parm7 (enter)

In theory, a tester would have to create 7! or 5040 tests. The problem is compounded even more if some of the parameters are optional. If the tester uses gray-box testing, by talking with the developer and understanding the parser algorithm, if each parameter is independent, only 7 tests may be required to test each parameter.

G17: Histograms

A histogram is a graphical description of measured values organized according to the frequency or relative frequency of occurrence. In Exhibit G.39, the table consists of a sample of 100 client/server terminal response times (enter key until a server response) for an application. This was measured with a performance testing tool.

The histogram in Exhibit G.40 illustrates how the raw performance data from the aforementioned table is displayed in a histogram. It should be noted that the design specification is for response times to be less than 3 seconds. It is obvious from the data that the performance requirement is not being satisfied and there is a performance problem.

G18: Inspections

Inspections are the most formal, commonly used form of peer review. The key feature of an inspection is the use of checklists to facilitate error detection. These checklists are updated as statistics indicate that certain types of errors are occurring more or less frequently than in the past. The most common types of inspections are conducted on the product design and code, although inspections may be used during any life-cycle phase.

Inspections should be short because they are often intensive; therefore, the product component to be reviewed must be small. Specifications or designs that result in 50 to 100 lines of code are usually manageable. This translates into an inspection of 15 minutes to 1 hour, although complex components may require as much as 2 hours. In any event, inspections of more than 2 hours are generally less effective and should be avoided.

Two or three days before the inspection, the producer assembles the input to the inspection and gives it to the coordinator for distribution. Participants are expected to study and make comments on the materials before the review.

The review is led by a participant other than the producer. Generally, the individual who has the greatest involvement in the next phase of the life cycle is designated as reader. For example, a requirements inspection would likely be led by a designer, a design review by an implementer, and so forth. The exception to this is the code inspection, which is led by the designer. The inspection is organized and coordinated by an individual designated as the group leader or coordinator.

The reader goes through the product component, using the checklist as a means to identify common types of errors as well as standards violations. A primary goal of an inspection is to identify items that can be modified to make the component more understandable, maintainable, or usable. Participants discuss any issues that they identified in the preinspection study.

At the end of the inspection, an accept or reject decision is made by the group, and the coordinator summarizes all the errors and problems detected and gives this list to all participants. The individual whose work was under review (e.g., designer, implementer, tester) uses the list to make revisions to the component. When revisions are implemented, the coordinator and producer go through a minireview, using the problem list as a checklist. The coordinator then completes management and summary reports. The summary report is used to update checklists for subsequent inspections.

G19: JADs

A JAD is a technique that brings users and development together to design systems in facilitated group sessions. Studies show that JADs increase productivity over traditional design techniques. JADs go beyond one-on-one interviews to collect information. They promote communication, cooperation, and teamwork among the participants by placing the user in the driver’s seat.

JADs are logically divided into phases: customization, session, and wrap-up. Regardless of what activity one is pursuing in development, these components will always exist. Each phase has its own objectives.

1. Customization — This phase is key to a JAD and largely consists of preparation for the next phase. Participants include the session leader and JAD analysts. The tasks include organizing the team, defining the JAD tasks and deliverables, and preparing the materials for the next JAD session.

2. Session — This phase consists of facilitated sessions in which the analysts and users jointly define the requirements and the system design. The session leader facilitates the session, and the analyst documents the results.

3. Wrap-Up — In this final phase, formal JAD outputs are produced. The facilitated session leader summarizes the visual and other documentation into a JAD document. The design results are fed back to the executive sponsor.

A given development effort may consist of a series of the three phases until the final requirements and design have been completed. When a project has multiple design activity (e.g., different portions of the overall design), a final wrap-up occurs at the completion of the design, at which point the design is reviewed as a whole.

G20: Orthogonal Array Testing

Orthogonal array testing, a statistical technique pioneered by Dr. Genichi Taguchi in manufacturing, helps in the selection of test cases to get a handle on the potentially enormous number of combination factors. It calculates the ideal number of tests required and identifies variations of input values and conditions; for example, it helps in the test selection process to provide maximum coverage with a minimum number of test cases.

Taguchi methods refer to techniques of quality engineering that embody both statistical process control (SPC) and new quality-related management techniques. Most of the attention and discussion of Taguchi methods have been focused on the statistical aspects of the procedure; it is the conceptual framework of a methodology for quality improvement and process robustness that needs to be emphasized.

An example is when the syntax of a command consists of three possible parameters in which there can be three possible values as follows.

       Command PARM1, PARM2, PARM3 (enter)
       PARMx = 1,2,3

In theory, a tester would have to create 33 or 27 test combinations, as shown in Exhibit G.41.

Applying orthogonal array testing (OATS), the technique selects test cases so as to test the interactions between independent measures called factors. Each factor also has a finite set of possible values called levels. In Exhibit G.41, there are three factors (PARM1, PARM2, and PARM3). Each has three levels (1, 2, and 3). The technique calls for the tester to locate the best fit of the number of factors and levels to the possible orthogonal arrays (found in most statistical texts). In Exhibit G.42, the orthogonal array with three factors and three levels is chosen. Each column in the array corresponds to a factor and each row corresponds to a test case. The rows represent all possible pairwise combinations of possible levels for the factors. Thus, only nine test cases are required, which demonstrates the power of the technique.

G21: Pareto Analysis

Pareto diagrams are a special form of a graph that points to where efforts should be concentrated. By depicting events or facts in order of decreasing frequency (or cost or failure rate, etc.), it allows for a quick separation of the “vital few” from the trivial many. The Pareto chart is more commonly known to information systems personnel as the 80-20 rule: that is, 20 percent of the causes make up 80 percent of the frequencies. A Pareto chart is a histogram showing values in descending order, which helps identify the high-frequency causes of problems so that appropriate corrective action can be taken. It is an organized ranking of causes of a problem by type of cause. The objective is to select the most frequent cause or causes of a problem to direct action to eliminate those causes.

The four steps in using a Pareto chart include the following:

1. Identify a problem area. One problem example is an excessive number of defects discovered during software testing.

2. Identify and name the causes of the problem. This is the most time-consuming step because it requires the collection of information from various causes. Causes of defects include the following: architectural, database integrity, documentation, functionality, GUI, installation, performance, and usability. For most problems, there is little need to identify more than 12 causes. When more than 12 causes can be identified, one approach is to select 11 causes and the 12th cause can be classified as “Other.” If the “Other” category becomes significant, then it may need to be broken down into specific causes.

   

   Exhibit G.43   Pareto Chart

3. Document the occurrence of the causes of the problem. The occurrences of the causes need to be documented. Samples from the defect-tracking database can be used to obtain these frequencies.

4. Rank the causes by frequency, using the Pareto chart. This involves two tasks: to count the problem occurrences by type, and to build a bar chart (or Pareto chart), with the major causes listed on the left-hand side and the other causes listed in descending order of occurrence.

In Exhibit G.43 there are eight defect causes. Approximately 1050 defects have been recorded. Of those, 750 are caused by functionality and database integrity. Thus, 20 percent of the causes account for 71 (or approximately 80 percent) of the frequency. In our example, functionality is the major cause, and database integrity is the second cause. Emphasis should be placed on eliminating the number of functional and database problems. One approach might be increased unit testing and reviews.

G22: Positive and Negative Testing

Positive and negative testing is an input-based testing technique that requires that a proper balance of positive and negative tests be performed. A positive test is one with a valid input, and a negative test is one with an invalid input. Because there typically are many more negative than positive tests, a suggested balance is 80 percent negative and 20 percent positive tests.

For example, suppose an application accepts stock market or mutual fund five-character symbols and then displays the respective stock or mutual fund name. An example of a positive test is “PHSTX,” which is the mutual fund symbol associated with a health science fund. If this symbol displayed some other fund, it would entail a positive test that failed.

Values that are not valid stock or mutual fund symbols are negative tests. Typically, a negative test produces an invalid error message. For example, if “ABCDE” is entered and an invalid error message is displayed, this is a negative test that passed.

Some considerations of negative testing are how much negative testing is enough and how do we anticipate unexpected conditions. Testing the editing of a single alphabetic character field can be complex. One negative test would be “(“and should be detected by the system. Should”)” be tested? How many other nonalphabetic characters should be tested? Unanticipated conditions are also sometimes difficult to detect. For example, “&” and “‘“ have special meaning with SQL. Should both of these be tested in every field?

G23: Prior Defect History Testing

With prior defect history testing, a test case is created or rerun for every defect found in prior tests of the system. The motivation for this is that defects tend to cluster and regress back to the original problem. Some causes include poor software configuration management procedures, poor coding and unit testing during defect repair, the tendency for bugs to cluster, and so on.

A defect matrix is an excellent tool that relates test cases to functions (or program units). A check entry in the defect matrix indicates that the test case is to be retested because a defect was previously discovered while running this test case. The absence of an entry indicates that the test does not need to be retested.

If this approach is not economical because a large number of defects have been discovered, a test case should be retested on or above a certain defined severity level.

G24: Prototyping

Prototyping is an iterative approach often used to build systems that users are initially unable to describe precisely. The concept is made possible largely through the power of fourth-generation languages and application generators. Prototyping is, however, as prone to defects as any other development effort, maybe more so if not performed in a systematic manner. Prototypes need to be tested as thoroughly as any other system. Testing can be difficult unless a systematic process has been established for developing prototypes.

The following sections describe several prototyping methodologies. They are presented to show the diversity of concepts used in defining software life cycles and to illustrate the effects of prototyping on the life cycle in general.

G25: Random Testing

Random testing is a technique in which a program or system is tested by selecting at random some subset of all possible input values. It is not an optimal testing technique, because it has a low probability of detecting many defects. It does, however, sometimes uncover defects that standardized testing techniques might not. It should, therefore, be considered an add-on testing technique.

G26: Range Testing

Range testing is a technique that assumes that the behavior of any input variable within a predefined range will be the same. The range over which the system behavior should be the same is first selected. Then an arbitrary representative from the range is selected and tested. If it passes, it is assumed that the rest of the values do not have to be tested.

For example, consider the following piece of code, which calculates the results Z from two input values X and Y:

If X and Y are positive integers ranging from 0 to 5 and X is greater than or equal to Y, there are 21 possible test cases, as depicted in Exhibit G.44.

Applying this technique has the potential of saving a lot of test generation time. However, it does have the limitation of the assumption that selecting an arbitrary input sample will produce the same system behavior for the rest of the inputs. Additional tests such as the conditions X and Y positive integers and Y greater than X need to be tested as well as the verification of square roots results; for example, we need to determine if the Z variable will accept fractional values as the result of the calculation or truncation (also see G4, “Boundary Value Testing”).

G27: Regression Testing

Regression testing checks the application in light of changes made during a development spiral, debugging, maintenance, or the development of a new release. This test must be performed after functional improvements or repairs have been made to a system to confirm that the changes have introduced no unintended side effects. Corrections of errors relating to logic and control flow, computational errors, and interface errors are examples of conditions that necessitate regression testing. Cosmetic errors generally do not affect other capabilities and do not require that regression testing be performed.

It would be ideal if all the tests in the test suite were rerun for each new spiral, but due to time constraints, this is probably not realistic. A good regression strategy during spiral development is for some regression testing to be performed during each spiral to ensure that previously demonstrated capabilities are not adversely affected by later development spirals or error corrections. During system testing after the system is stable and the functionality has been verified, regression testing should consist of a subset of the system tests. Policies need to be created to decide which tests to include.

In theory, the reliability of a system that has been modified cannot be guaranteed without a full regression test of all tests. However, there are many practical considerations:

1. ■ When defects are uncovered, additional regression tests should be created.

2. ■ A regression test library should be available and maintained as it evolves.

3. ■ There should be a methodology of isolating regression tests that focus on certain areas (see retest and defect matrices).

4. ■ If the overall architecture of a system is changed, full regression testing should be performed.

5. ■ Automated testing with capture/playback features should be strongly considered (see Section 6, “Modern Software Testing Tools”).

G28: Risk-Based Testing

The purpose of risk management testing is to measure the degree of business risk in an application system to improve testing. This is accomplished in two ways: high-risk applications can be identified and subjected to more extensive testing, and risk analysis can help identify the error-prone components of an individual application so that testing can be directed at those components.

Risk analysis is a formal method for identifying vulnerabilities (i.e., areas of potential loss). Any area that could be misused, intentionally or accidentally, and result in a loss to the organization is a vulnerability. Identification of risks allows the testing process to measure the potential effect of those vulnerabilities (e.g., the maximum loss that could occur if the risk or vulnerability were exploited).

Risk-based testing is a technique in which test cases are created for every major risk factor that has been previously identified. Each condition is tested to verify that the risk has been averted.

G29: Run Charts

A run chart is a graphical representation of how a quality characteristic varies with time. It is usually a line graph that shows the variability in a measurement or in a count of items. For example, in Exhibit G.45, a run chart can show the variability in the number of defects detected over time. It can show results from a sample of a population or from 100 percent.

A control chart, a special form of run chart, places lines on the chart to represent the limits of permissible variability. These limits could be determined by a design specification or an agreed-upon standard. The control limits are frequently set to show the statistical limit of variabilities that could be due to a chance occurrence. This is calculated by using the averages and range of measurement from each sample of data. Control charts are not only used as an alarm when going outside the limits, but also to examine trends occurring within the limits. For example, if the sequence of ten measurements in Exhibit G.45 is shown to fall above the expected average, it can be assumed that this is not due to mere chance and, therefore, an investigation is in order.

G30: Sandwich Testing

Sandwich testing uses top-down and bottom-up techniques simultaneously and is a compromise between the two. The approach integrates from the top and bottom at the same time, meeting somewhere in the middle of the hierarchical control structure. The meeting point in the middle is defined by the program structure.

It is typically used on large programs but is difficult to justify on small programs. The top level of the hierarchy usually includes the user interfaces to the system, which requires stubs to mimic business functions. The bottom level includes primitive-level modules that require drivers to simulate lower-level modules.

G31: Statement Coverage Testing

Statement coverage is a white-box technique that ensures that every statement or line of code (LOC) is executed at least once. It does guarantee that every statement is executed, but it is a very weak code coverage approach and not as comprehensive as other techniques, such as branch coverage, where each branch from a decision statement is executed.

Consider the following small program, which reads records from a file and tallies the numerical ranges of a field on each record to illustrate the technique.

G32: State Transition Testing

State transition testing is a testing technique in which the states of a system are first identified. Then a test case is written to test the triggers or stimuli that cause a transition from one condition to another state. The tests can be designed using a finite-state diagram or an equivalent table.

Consider the following small program, which reads records from a file and tallies the numerical ranges of a field on each record to illustrate the technique.

G33: Statistical Profile Testing

With statistical profile testing, statistical techniques are used to develop a usage profile of the system. Based on the expected frequency of use, the tester determines the transaction paths, conditions, functional areas, and data tables that merit focus in testing. The tests are, therefore, geared to the most frequently used part of the system.

G34: Structured Walkthroughs

Structured walkthroughs are more formal than the code-reading reviews. Distinct roles and responsibilities are assigned before the review. Preview preparation is greater, and a more formal approach to problem documentation is stressed. Another key feature of this review is that it is presented by the producer. The most common walkthroughs are those held during design and coding; however, recently they have been applied to specifications documentation and test results.

The producer schedules the review and assembles and distributes input. In most cases, the producer selects the walkthrough participants (although this is sometimes done by management) and notifies them of their roles and responsibilities. The walkthrough is usually conducted with less than seven participants and lasts no more than 2 hours. If more time is needed, there should be a break or the product should be reduced in size. Roles usually included in a walkthrough are producer, coordinator, recorder, and representatives of user, maintenance, and standards organizations.

Although the review is opened by the coordinator, the producer is responsible for leading the group through the product. In the case of design and code walkthroughs, the producer simulates the operation of the component, allowing each participant to comment, depending on that individual’s area of specialization. A list of problems is kept, and at the end of the review, each participant signs the list, or other walkthrough form, indicating whether the product is accepted as is, accepted with recommended changes, or rejected. Suggested changes are made at the discretion of the producer. There are no formal means of follow-up on the review comments. If the walkthrough review is used for products throughout the life cycle, however, comments from past reviews can be discussed at the start of the next review.

G35: Syntax Testing

Syntax testing is a technique in which a syntax command generator generates test cases based on the syntax rules of the system. Both valid and invalid values are created. It is a data-driven black-box testing technique for testing input data to language processors, such as string processors and compilers. Test cases are developed based on rigid data definitions. The valid inputs are described in Backus–Naur Form (BNF) notation.

The main advantage of syntax testing is that it ensures that no misunderstandings about valid and invalid data and specification problems will become apparent when employing this technique.

G36: Table Testing

Table testing is a technique that tests the table, which is usually associated with a relational database (the same approaches can be applied to arrays, queues, and heaps). Tables usually come in two forms: sequential and indexed. The following are general tests that need to be performed against tables:

1. Indexed Tables:

a.   Delete the first record in the table.

b.   Delete a middle record in the table.

c.   Delete the last record in the table.

d.   Add a new first record in the table.

e.   Add a new middle record in the table.

f.   Add a new last record in the table.

g.   Attempt to add a duplicate record.

h.   Add a record with an invalid key, for example, garbage in the key field.

i.   Change the key fields on a existing record; for example, change an order number.

j.   Delete a nonexisting record; for example, enter a delete key that does not match table entries.

k.   Update and rewrite an existing record.

2. Sequential Tables:

a.   Attempt to delete a record from an empty table.

b.   Read a record from an empty table.

c.   Add a record to a full table.

d.   Delete one record from a one-record table.

e.   Read the last record.

f.   Read the next record after the last record.

g.   Scroll sequentially through the table.

h.   Insert an out-of-sequence record.

i.   Attempt to insert a duplicate record.

G37: Thread Testing

Thread testing is a software testing technique that demonstrates key functional capabilities by testing a string of program units that accomplishes a specific business function in the application.

A thread is basically a business transaction consisting of a set of functions. It is a single discrete process that threads through the whole system. Each function is tested separately, then added one at a time to the thread. The business transaction thread is then tested. Threads are in turn integrated and incrementally tested as subsystems, and then the whole system is tested. This approach facilitates early systems and acceptance testing.

G38: Top-Down Testing

The top-down testing technique is an incremental approach in which the high-level modules or system components are integrated and tested first. Testing then proceeds hierarchically to the bottom level. This technique requires the creation of stubs. When a module or system component is tested, the modules or components it invokes are represented by stubs, which return control back to the calling module or system component with a simulated result. As testing progresses down the program structure, each stub is replaced by the actual code it represents. There is no rule that specifies which module to test next; the only rule is that at least one of the modules or system component-calling modules must have been tested previously.

Top-down testing allows early discovery of major design flaws occurring at the top of the program, because high-level functions and decisions are tested early, and they are generally located at the top of the control structure. This verifies the program design early. An early prototype or initial design facilitates early demonstrations. Because the menus are often at the top of the control structure, the external interfaces can be displayed early to the user. Stubs need to be created, but are generally easier to create than drivers. On the other hand, critical low-level modules or system components are not tested until late in the process. In rare cases, problems with these critical modules or system components may force a redesign.

G39: White-Box Testing

White-box testing, or structural testing, is one in which test conditions are designed by examining paths of logic. The tester examines the internal structure of the program or system. Test data are driven by examining the logic of the program or system, without concern for the program or system requirements. The tester has knowledge of the internal program structure and logic, just as a mechanic knows the inner workings of an automobile. Specific examples in this category include basis path analysis, statement coverage, branch coverage, condition coverage, and branch/condition coverage.

An advantage of white-box testing is that it is thorough and focuses on the produced code. Because there is knowledge of the internal structure or logic, errors or deliberate mischief on the part of a programmer has a higher probability of being detected.

One disadvantage of white-box testing is that it does not verify that the specifications are correct; that is, it focuses only on the internal logic and does not verify the logic to the specification. Another disadvantage is that there is no way to detect missing paths and data-sensitive errors. For example, if the statement in a program should be coded “if |a–b| < 10” but is coded “if (a–b) < 1,” this would not be detectable without specification details. A final disadvantage is that white-box testing cannot execute all possible logic paths through a program, because this would entail an astronomically large number of tests.