CHAPTER 3

Testing Techniques for Markup Applications

In my experience the majority of automated testing for markup applications either uses HTTP requests and responses using a protocol library such as urllib or a browser-based automation library such as Selenium. Both approaches are useful and can be effective.

A web browser can be used as a simple way to quickly view the content returned for a given xHTML device. The browser will need an add-in to incorporate sufficient device headers to convince the web site that the request is from a particular mobile device. Similarly, WML can be rendered in Firefox by using another browser add-in. More information is available in the section titled “Utilities to help with testing browser applications.”

3.1 GETTING STARTED WITH TEST AUTOMATION

Let us start with an overview of how you can implement your first automated script for testing browser applications:

• Implement commands to send a HTTP request and receive the response.
• Add some basic validation of the HTTP response in order to determine whether you are getting what you want, and not getting what you do not want:
  • Does the HTTP status code equals 200? (OK.)
  • Is the content type what you expected?
  • Is the content length appropriate? (You may be able to use a single exact value or specify a range of acceptable values.)
• Add pattern-matching (e.g., to match the word “Results” followed by two numbers, followed by “of about”) for the UK English search results on www.google.co.uk. See screenshot for an example of the search results page (Figure 3.1).
• Add a user-agent string that matches one from a specific mobile phone.
• If necessary, add other HTTP headers such as:
  • Accept,
  • X-WAP.

 

FIGURE 3.1: Google search results.

Consider adding code to detect failures, e.g., error messages that should not be in the responses. By detecting failures explicitly the tests will be more reliable (harder to fool) and errors can be handled sooner, rather than the test meekly waiting for the “expected result” until a timeout occurs.

3.2 EXAMPLES WRITTEN IN PYTHON

The examples here are written in Python, a flexible programming language that is easy to experiment with interactively as well as being able to handle large-scale programs. Several other programming languages are able to provide similar capabilities, so try yours if you don't find Python suitable.

EXAMPLE 3.1: Four lines of Python to retrieve a web response.

With four lines of Python, shown in Example 3.1, we are able to retrieve the contents of Google's mobile search homepage for the UK site. Note: the numbers of the bullet-points refer to the numbers in brackets from the preceding example Python code.

 

1. import urllib does all the hard work of making the http request and returning the response, e.g., the library automatically handles HTTP redirection.
2. Create a request object, using one of the methods provided by urllib.
3. Make the request to retrieve the contents of the mobile search page, and assign the response to the response object.
4. The content of the response is returned using the read() method. We can now test the content directly by using standard Python functionality.

 

So with four simple lines of code we have the page content. We can also obtain more information, like the HTTP response headers by calling response.info()

EXAMPLE 3.2: Display the HTTP response headers for WML.

And individual headers using response.info().typeheader

EXAMPLE 3.3: Display the content type response header for WML.

Here you may notice something strange: the content type indicates the content is in WML, rather than xHTML or HTML. Google's search engine seems to be defaulting to the oldest markup language, rather than the most popular one—why is that?

Early mobile wireless devices used WAP requests and expected WML responses. Later phones were often able to support both WML and xHTML, while newer phones support xHTML and HTML. Since the search engine does not have any indication of what markup language the requester wanted, it sends the earliest version. Google search is able to deliver the correct markup provided it receives sufficient “clues” in the HTTP request.

Two key HTTP headers generally provide enough information for the search engine to deliver appropriate content for a given device. By adding these headers we should be able to “fool” the search engine into returning the content it would return to the equivalent phone model. Let us go through these headers one by one:

The first header to consider is the “Accept” header. This is part of the HTTP standard and is used by the requester to tell the server what types of content it is able to use. For a desktop web browser the Accept header may be set to

 

 

This “accept” header tells the server the image file formats the browser can display, and then a catch-all for other content types, e.g., for text/html.

For a mobile browser that accepts xHTML, the Accept header will generally be set to:

 

As indicated with (1) in Example 3.4, in Python we can add this header using:

EXAMPLE 3.4: Adding the accept header for xHTML.

EXAMPLE 3.5: Displaying the HTTP response headers for xHTML.

And using response.info().typeheader this time shows the response is in xHTML—phew!

EXAMPLE 3.6: Display the content type for xHTML.

At the time this example was created, September 2007, Google was promoting the “Google Maps” mobile application to users who had suitable phones. I wanted to write a simple test that checks whether the link is available on the xHTML homepage. As the link is only offered to users with suitable phone models, we need our test to pretend it is one of those phones. The User-Agent HTTP header is what we need to achieve this.

Here is an example of a user-agent string from a Nokia 6230 phone:

 

 

We can add this to our request by adding another header (shown in bold and indicated with (1) in Example 3.7:

 

EXAMPLE 3.7: Adding a user-agent string to emulate a Nokia 6230.

3.2.1 A Test to Detect if Google Maps Is offered to Mobile Users

Let us start by using a simple search to find out whether Google Maps is mentioned anywhere on the page.

EXAMPLE 3.8: Using string search to find out if the content contains Google Maps.

If the string is not found content.find() returns -1.

EXAMPLE 3.9: The String search returns -1 if the string is not found.

So, a simple string search is enough to get us started, and here indicates that the string “Google Maps” is contained in the returned content. We could now write more involved “string searches,” e.g., to find and extract the URL for the download and make sure the page does not have multiple links to Google Maps, etc. However, Python offers a better approach using the very powerful regular-expression tools.

3.2.2 Using Regular Expressions in Our Test

Four more lines of Python code are enough to perform the regular expression match and return the link to the download page.

EXAMPLE 3.10: Using a regular expression to get the download link for Google Maps.

Here are the four steps required:

 

1. Import the re regular expression module (provided as a standard Python library module).
2. Define and compile the regular expression. The .* characters mean: match any characters between the href and the string “Google Maps”.
3. Perform the search on the content of the web results. If matches are found the m variable will be assigned to point to the set of matches.
4. m.group() returns all the matches, here there is only one match, which is displayed on the next line. The link is a relative link: /gmm followed by some parameters that are useful for tracking the promotion.

 

We could easily refine this code to extract, and even download, the Google Maps software.

3.2.3 Combining XML With Regular Expressions

Both WML and xHTML return XML documents; therefore we can use XML processing techniques to locate content of interest. The following example demonstrates how to use standard xml Python modules to match a string or even a regular expression and return the link if it exists.

EXAMPLE 3.11: getlink.py.

3.2.4 Using XPATH in Our Tests

There is a powerful free Python module called Amara that can be used to address the xml structure using an intuitive dot addressing, e.g., html.body.div. Amara also supports XPATH expressions, e.g., //div[0].

Here is an example of using Amara to test whether the Google Maps link is available. The link is returned if Google Maps is found in the text of a link.

EXAMPLE 3.12: Using Amara to extract the Google Maps link.

Example 3.12 is a little more involved, but still relatively easy to follow:

 

1. Use Amara to parse the web result. In this example, the code uses a saved example of the search results. To read the result from the web, simply replace the filename with the full URL starting with http://
2. getHrefFromXML is a helper method, which we can use many times to match various regular expressions.
3. These two lines are commented out (using the # character). These are examples of debugging the helper method and these lines are generally removed once we have debugged the method.
4. By using item.a.xml_children[0] we restrict the match to one part of the XML structure in the response.
5. Return the match, if found. Otherwise the helper method will return None (a Python reserved word that we use to indicate no match was found).
6. This is a standard convention in Python to execute the subsequent code iff the script is being run directly (rather than as part of a library).
7. Test the helper function by searching for “Google Maps”.

3.3 SUMMARY OF THE EXAMPLES IN PYTHON

These sample Python scripts have demonstrated the various ways we can process xHTML content to:

 

• Make a basic HTTP request.
• Set the content-type to control the markup language returned by the web server.
• Set the user-agent to emulate a particular phone model.
• Parse the returned xHTML content to determine whether the Google Maps for Mobile link is provided for particular phone models.

 

3.4 BUILDING ON YOUR FIRST AUTOMATED SCRIPTS

We can build on these basic scripts to automate more of our mobile browser testing.

3.4.1 Data-Driven Tests

We can make the tests “data-driven” as follows:

 

• Define a of input parameters. Consider parameters such as:
  • User-agent string,
  • Accept header,
  • URL,
  • Input parameters,
  • HTTP verb (GET or POST), and
  • Expected results, e.g., should there be a link for ‘Google Maps’? There may be specific variations for particular devices, e.g., different download links.
• Read these from an external source.
• Return the results, e.g., as a file for later checking. Some basic checks, e.g., for the contenttype may be performed by the program.

 

3.4.2 Obtaining Metadata to Drive Our Tests

Metadata facilitates our automated tests, e.g., user-agent strings, accept headers, etc. Data sources include:

 

• commercial,
• WURFL,
• local data, and
• http://www.pycopia.net/webtools/headers (which captures the headers from the web browser on the device you are using and can email them to you, to save copying and pasting).

 

Find ways to rate and weigh the data quality, as data is often incomplete, contradictory, or wrong. You may decide it's worth filtering and merging useful data into a common pool.

3.4.3 Using Metadata

Once you have collected, filtered and merged your metadata it can be made available as a common source of data for your tests (and possibly also for your production systems). Figure 3.2 illustrates a single example of metadata for a Nokia 6230i device.

Do stuff with device data…

FIGURE 3.2: Combining sources of data with an example of data for a phone.

3.4.4 Test Using Carrier Networks

Carrier networks can sometimes affect the results returned, and sometimes even corrupt the content. If you have a suitable data connection (e.g., using a GPRS modem or a phone connected as a data modem) you should be able to run the tests over carrier networks and compare the responses returned over each carrier. Generally we expect all the responses to be identical (for the patterns and content we expect to receive). If differences occur, potentially, a given carrier's network/infrastructure has modified the content. The differences should then be investigated and assessed in terms of the impact they have on users.

Your test scripts should:

 

• Record the carrier network used (this may be as simple as using a user-specified string, or more complex, e.g., by querying the modem using on of the common GPRS AT commands : AT+COPS? for the currently connected carrier network before running the tests);
• Compare the response against a known reference page (possibly downloaded over your wired network); and
• Highlight differences, e.g., using a diff program for a user to compare the differences. Of course you can write more sophisticated comparison algorithms to reduce the need for people to get involved.

 

Note: One of the appendices includes information on how to configure a suitable data connection in Linux.

Networks and data plans vary and can affect the functioning of your applications. While you could test using an application on a phone, the problems may be hard to isolate from other issues related to the phone(s) you are using to test. A good way to isolate carrier and data plan issues is to run a custom test program on a computer with a wireless data modem, as Figure 3.3 shows.

FIGURE 3.3: Testing the carrier network.

3.4.5 Timing the Request/Response Pair

One of the key frustrations of users for mobile wireless applications is how slow they are. Virtually all modern programming languages offer the ability to time how long things take to happen. If you start a timer just before making the HTTP request and stop it as soon as the HTTP response has been received, you will have a good idea of how long the OTA part of the transaction takes. If you choose to run similar tests for a range of carriers you can obtain a rough idea of the speed of each carrier for your local network conditions. Note: the speed varies significantly as local conditions change. Factors include:

 

• The protocol used to connect (e.g., GPRS is generally much slower than 3G),
• The number of active connections to the operator's local base station,
• The distance from the base station,
• The physical topology,
• The performance of the mobile wireless device, and
• The weather.

 

Measuring the end-user experience is much more involved and outside the scope of the current material.

3.4.6 Implementing Rule-Based Tests

In our experience, certain patterns of xHTML cause problems for particular phone models. Sometimes the issue is size of the response, for others tags such as the bold tag cause the contents to be rendered poorly, some handsets have limitations on the image formats they can handle, one phone even makes the text larger when the tag specifies the font size should be small, etc.

The effect ranges from the page not being displayed at all to minor rendering issues.

We do not want our users to have a poor user-experience, so we want to ensure our content will not trigger issues on their phones. At one point we created individual tests to detect whether particular content was appropriate for a given model of phone. However, that work was time-consuming and did not scale well. Therefore we designed and implemented a rule-based engine that can query URLs while emulating a wide range of phone models. The responses are then checked using rules which detect any violation of the known issues for that model of phone.

We gather issues from a variety of sources, e.g., from bug reports, server logs, manual testing, etc; identify them (in terms of which devices are affected and when the issue occurs); quantify them (in terms of severity, impact, likelihood, etc.); and write rules to detect the issue.

We simulate the affected devices using a mobile device database (loosely based on WURFL, an open-source effort to collect mobile device characteristics). We make requests to a variety of web sites that should work for mobile devices and apply the rules to the response received. If a rule is broken the software reports an error, i.e., if the content would cause an issue for the emulated device.

Generally we are able to manually verify whether a bug really occurs by testing with a real phone. This helps us to remove false positives from our system.

Our developers are then able to modify the application software to prevent the issue from occurring. Their changes are generally also tested manually, e.g., for aesthetics and to ensure the content renders correctly on the affected devices.

3.4.7 Probe Servers

Sometimes we are left with a number of open questions, such as:

 

• What are the capabilities various devices?
• What content causes problems?
• What is the signature of each device?

 

One way to obtain the answers is to create a “Probe server” that interacts with devices to obtain the answers we need. The tests need to be unambiguous for the tester, particularly in terms of their ability to provide accurate answers. If the tester is confused on what the “correct” answer is then the tests are likely to take longer while the quality of the data may also be compromised. Also, try to reduce the time required for each test. I am aware of one commercial vendor that requires a tester to manually execute more than 1,000 distinct tests per device in order to identify the device's characteristics, which take several days to execute. I don't envy their testers!

3.4.8 Strengths and Weaknesses of Rule-Based Testing

Strengths:

 

• Great for regression testing and for testing new applications that need to deliver content to mobile devices.
• Rules are generally relatively easy to codify, and easy to execute automatically.
• Additional rules can be added to check for accessibility using internationally recognized guidelines from the Web Content Accessibility Guidelines (WCAG, http://www.w3.org/TR/WAI-WEBCONTENT/).
• It can test some aspects of user experience, e.g., to detect consistency across multiple web sites for a large companies. Potentially, CSS can be checked dynamically (as it can be affected by client-side scripting, the model of web-browser, etc.).

 

Weaknesses:

 

• Relies on the quality of the mobile device database, which has proven to be inaccurate.
• Each issue needs to be identified, quantified and coded. The coding tends to require some technical understanding of the underlying markup language, regular expressions and/or XPATHs, and Java.
• False positives need to be tested manually (and false negatives need to be fixed too).
• Does not currently simulate user-input.
• Does not test navigation, or scripting (e.g., JavaScript).

 

3.4.9 A Complementary Tool to Rule-Based Tests

One way to learn about the characteristics and issues for mobile phones is to use the phone to interact with a known set of web pages that contain various test cases, e.g., to determine which images are supported. If the web pages are interactive, then the user can provide feedback online while executing the tests (e.g., to confirm whether an image has been displayed correctly or not). The resulting data can then be codified into rules for that model of phone.

Furthermore, by running the rule-based checker against the test web pages the accuracy of the rule-based checker can be verified.

3.4.10 Is Appropriate Content Being Served?

Large, sophisticated international web sites need to deliver content that is appropriate for each user. Factors that affect the selection of content include things like:

 

• The device being used (e.g., an xHTML phone, an iPhone, a desktop browser) and its capabilities to display content.
• Whether scripting is supported on the client?
• The location of the user (which can affect the selection of localized content and the human language returned).
• User preferences, which may override default language and content selections.

 

Sometimes we redirect an initial request to a more suitable user interface, e.g., from the general search to the mobile search, based on the capabilities of the device.

Automated tests can be used to test all aspects of these requirements to varying degrees of accuracy. For instance the device being used can be emulated quite easily using HTTP headers. Implementing support in our tests to process scripting support is significantly harder. If your tests need to process scripts, consider using third-party open-source libraries such as HtmlUnit.

Location can be inferred from data such as source IP address and other factors such as phone-specific HTTP headers.

User preferences are often stored in cookies and cookie content can be added relatively easily to HTTP requests. Some more sophisticated cookie strategies require more involved programming to mimic the behavior of the way a device handles them.

3.5 TIPS WHEN IMPLEMENTING AUTOMATED TEST SCRIPTS

 

• Pick a simple web page that does not change, consider running a local web server to serve test pages initially.
• Use a network traffic analyzer to record the requests and responses.
• Get a trustworthy HTTP protocol reference (see Appendix A for some examples).
• Try to download pages from a web server that captures the HTTP headers. Access that web server from several different mobile phones and save the captured data (see Appendix A for an example of such a server).

 

3.6 TEST TOOLS FOR BROWSER-BASED APPLICATIONS

3.6.1 Using Web-Testing Tools

For xHTML, the content is sufficiently similar to HTML to allow us to use many of the generic automated test tools, provided the tools offer the ability to set and read the HTTP request and response headers.

HttpUnit and HtmlUnit enable Java developers familiar with the JUnit test framework to create similar tests as easily as using Python (except Java does not provide an interactive development capability). HtmlUnit is more capable and, as mentioned earlier, includes support for testing the JavaScript scripting language.

3.6.2 “Mobile Readiness” Tools

Various software tools and web sites provide the ability to test the “readiness” of a web site for mobile wireless devices. For instance, http://ready.mobi/launch.jsp?locale=en_EN allows a web site to be checked online in terms of readiness for mobile devices. There are other similar sites and services available; however, this one has more of a testing focus. It implements the w3c mobileOK basic tests http://www.w3.org/TR/mobileOK-basic10-tests/ and is able to simulate a variety of common handsets.

3.6.3 Utilities to Help With Testing Browser Applications

Browser add-ons such as: wmlbrowser, web developer, user agent switcher, and modify headers (all for Firefox) make manual testing significantly easier. All of these add-ons can be installed directly from Firefox from the tools menu. On Microsoft Windows the menu option is “Extensions” on Linux it is “Add-ons.”

XML tools, such as Oxygen (http://www.oxygenxml.com/), a commercial product, reduce the challenges of working with XML.

Firebug is a stunningly useful extension for Firefox that provides incredible analysis and debugging tools. Features range from displaying the XPATH of an element when the mouse is “hovered-over” page elements on an xHTML web page, to interactive debugging of JavaScript.

Another important tool for testing browser applications is a network traffic analyzer that helps decode the requests and responses actually sent between your machine and the server, rather than what you think or hope is being sent.