One of the indisputable things when building and delivering any product is its quality. End users of a product will not settle for anything less than superior when it comes to product quality. In this chapter, we will dive deep into how testing and test automation help achieve this level of quality in a software product. The first few pages will introduce the reader to testing and test automation. Later in the chapter, we will dive deeper into the subject of test automation.
Quality is everyone’s responsibility in a team, and this chapter provides various practical pointers to help establish that collaboration. Additionally, we will see how automated tests add another layer of complexity to a project and help understand the best practices to accomplish coherent test automation. We will also look at the lineup of development and test automation processes to provide a reliable and bug-free product experience for customers.
Here are the key topics that you will learn by the end of this chapter:
Testing as an activity is critical to delivering a trustworthy product and forms a strong foundation for building reliable test automation. So, being a good tester makes you a more effective test automation engineer.
Software testing is an indispensable task in any software development project that is mainly done with the goals of validating the specified product requirements and finding bugs. These bugs can be functional or non-functional in nature. Functional bugs include deviations from the specified requirements or product specifications. Usually, non-functional issues are performance-based or security-based. The primary goals of testing are usually interwoven at multiple levels but can be broken down at a high level as follows:
Now that we have seen a quick introduction to testing, let us understand why testing is so critical.
Even though software’s quality is initially laid out by the design and architecture, testing is the core activity that gives stakeholders much-needed confidence in the product. By verifying the behavior of the product against documented test cases, the activity of testing helps uncover bugs and address other design issues promptly. By preventing and identifying bugs early in the software development life cycle, testing helps both the engineering and business teams to increase customer satisfaction and reduce overall operating costs. The valuable insights that the team derives from the repeated testing of the product can be further used to improve the efficiency of the software development process. Therefore, it is imperative to conduct testing for achieving the established goals of a successful product launch.
Testing is not just executing the documented test cases. There are a multitude of tasks involved in testing, as follows:
Depending on the team size, capacity, and structure, some or all of these activities must be performed for a successful release of a software product.
In the following diagram, we get a comprehensive view of all the deliverables involved in software testing:
Figure 1.1 – Testing deliverables
As you can see in the preceding diagram, testing encompasses a wide variety of deliverables resulting from many cross-functional activities. Next, let’s look at some unique demands for testing in the world.
Unlike a traditional waterfall model, in the Agile world, it is recommended that each user story or feature has the right balance of manual and automated tests before calling it done. This arguably slows down the team, but it also drastically reduces the technical debt and improves code quality. You will notice that scaling the software becomes easy, and there is a significant decrease in reworking as the automated tests increase. This saves huge amounts of time for the engineering team in the long run. Another common occurrence in an Agile setup is developers writing automated tests. Usually, developers own unit tests, and quality engineers write other types of tests. Sometimes, it is productive for the developers to write automated test code while the quality engineers focus on other testing tasks. One of the most important takeaways from the Agile world is that quality is everyone’s responsibility on the team. Test engineers keep the communication and feedback going constantly between the product and the software engineers, which, in turn, results in an excellent team culture. This also opens up early discussions on the necessary tools and infrastructure needed for testing.
Since the Agile environment aims to deliver the simplest possible product to the customer as quickly as possible, the test engineers focus on the most necessary set of test cases for that product to be functionally correct. In each increment, the test engineers create and execute a basic set of test cases for the features being delivered to the customer. Test engineers are constantly working with the product, developers, and in some cases, with customers to keep the stories as simple and concise as feasible. Additionally, the Agile landscape opens doors to automating the entire software development life cycle using the principles of continuous integration, which demands a major shift in the test engineer’s mindset. This demands excellent communication skills and fluent collaboration from the test engineers.
Often, a test engineer’s output is measured by the number and quality of defects they find in the software application. Let us examine some key aspects of defect management next.
A defect is primarily a deviation from the business requirement, and it does not necessarily mean there is an error in the code. The analyzing, documenting, and reporting of defects in the product is a core activity for the testing team. It is essential to set and follow standard templates for reporting defects. There are a variety of tools in the market that help with test case and defect management. Maintaining a good rapport with all the engineers on the team goes a long way in making the defect reporting and resolving process much smoother. Usually, testers have to put in as much information as possible in the defect log to assist developers in reproducing the defects in various environments as needed. This also helps in categorizing the defects correctly. There are cases when the defect still cannot be reproduced consistently, and the engineer and tester have to pair up in order to spot the failure in the application precisely. It is important to file high-quality functional defects and follow up when necessary to get them fixed on time. So, the testing team is primarily responsible for the defect management process and keeping the higher management updated on any high-severity defects that might affect the product release.
Defect versus bug
A defect is a deviation from the expected behavior. It can be a missing, incorrect, or extra implementation. In comparison, a bug is a programming error that causes the software to crash, produce incorrect results, or perform badly.
Besides the number of bugs being identified, it is also crucial to know when a defect is identified in the project. Next, let’s explore the effects of shifting testing early or late in the development life cycle.
It is evident that there are a lot of benefits that can be reaped by shifting the testing and quality efforts, in general, earlier on in the development life cycle. This is termed the Shift-Left approach. In this approach, quality engineers are involved right from the inception of the project in early discussions about the design and architecture. They start working on deliverables such as the test plan and test cases in parallel with developers. This approach builds a quality-first mindset for the entire team and ensures quality is built right into the vein of every activity in the development life cycle.
The Shift-Right approach is an extension of Shift-Left, where the testing team’s responsibilities are stretched further into the release of the product and maintenance. Test engineers, with their deep knowledge of the product, assist with the implementation and help with testing and monitoring in production. Additionally, the Shift-Right approach has some test engineers support performance, load, and security testing. There is also good scope for test automation to address quality issues under real-world conditions within this approach. Both these approaches are equally important. While the push for Shift-Left has been happening for quite some time now, the Shift-Right approach is relatively new and is extending the tester’s involvement in obtaining and testing actionable production data in safer ways.
This leads us to the question of how to incorporate quality early on, and DevOps processes have helped enormously in that regard. Next, let’s look at how quality and DevOps blend together to deliver organizational value.
The previous section about the Shift-Right approach leads us right into the area of DevOps and how it has accelerated product development and changed the quality process. DevOps strives continuously to align high-performing engineering teams to business value in the most efficient way. Quality is a core component in each of the DevOps processes. DevOps attempts to automate each and every task in the product delivery, right from building the code to deploying the application to production for customers to use. This adds further emphasis on quality. Since the whole process is automated, feedback at every checkpoint is crucial. A predetermined set of units, functional, and integration tests executed at the right times in the deployment pipeline act as a gate to the production deployment. At times, the manual involvement of test engineers will be required for debugging test failures and for specific types of testing.
In the DevOps world, it is essential to maintain nimbleness in testing activities and deploy the right type of testing resources when and where needed. We will dive deep into how test automation helps in continuous integration and continuous delivery in Chapter 9, CI, CD, and Test Automation. For now, it is good to grasp the significant role that testing plays in ensuring a successful DevOps implementation.
So far, we have seen various aspects of testing, and it is undoubtedly a demanding activity. Let’s examine some challenges that test engineers face on a daily basis.
Before diving into the world of test automation, it is vital to understand the common challenges faced in the testing world. Building a solid manual testing foundation is paramount to developing a sound test automation strategy. A clear and concise testing process acts as a strong pillar for building reliable automation scripts. Some of the challenges faced regularly by the testing team are outlined as follows:
Next, let’s look at how testing early and often helps overcome some of the obstacles in the testing journey.
Usually, an enterprise software application contains hundreds, if not thousands, of components. Software testing has to ensure the reliability, accuracy, and availability of each of these components. So, it is inevitable to emphasize the importance of the common quality industry term, Test early and test often:
Table 1.1 – Importance of testing early and testing often
The following excerpt highlights the cost of fixing bugs later in the product development cycle. As Kent Beck explains in his book, Extreme Programming Explained, “Most defects end up costing more than it would have cost to prevent them. Defects are expensive when they occur, both the direct costs of fixing the defects and the indirect costs because of damaged relationships, lost business, and lost development time.”
The common arguments for not testing early are “We don’t have the resources” or “We don’t have enough time.” Both of these are results of not examining the risks of the project thoroughly. It is a proven fact that the total amount of effort is greater when testing is introduced later on in the project. On the tester’s part, it is important to reduce redundancy and constantly think about increasing the efficiency of their tests. Test-Driven-Design (TDD) is a common approach to practice Test Early, Test Often. Testing processes have to be fine-tuned and adhered to strictly for this approach to be successful. Testing can have a strategic impact on the quality of the product if introduced early and done often and efficiently. The Agile test pyramid (which is discussed, in detail, in Chapter 2) can act as a guide to strategically categorize and set up different types of tests.
So far, we have dealt with a range of concepts that help us acquire a well-rounded knowledge of testing. We looked at the importance of testing and how crucial it is to integrate it with every facet of the software development process. With this background, let us take on the primary topic of this book, test automation.
Imagine an engineering team adding one or two software engineers every quarter. This team wants to delight the customer by delivering more features every sprint. Even though they have one or two quality engineers on the team to test all of the new features, they notice that the faster they try to deliver code, the higher the number of regression bugs introduced. The manual testing of the features just isn’t enough. They want certain core test scenarios to be executed repetitively and, often, when the changes are introduced. This is where test automation helps tremendously. It is always better to get started with test automation before feeling the agony of a high-severity bug in production or a catastrophic incident happening due to the lack of timely testing.
Testing is definitely not a one-time activity and must be done any time and every time a change is introduced into the software application. The longer we go without testing an application, the higher the chances of failure. So, continuous testing is not an option but an absolute necessity in today’s Agile software engineering landscape. Manual testing involves a tester executing the predefined test cases against the system under test as a real user would. Automated testing involves the same tests to be run by a script, thereby saving the tester’s valuable time so that they can focus on usability or exploratory testing. When done right, automated tests are more reliable than a manual tester. More importantly, it provides more time for the tester to draw valuable insights from the results of the automated tests, which further aids in increasing the test coverage of the software application as a whole.
Test automation is one of the chief ways to set up and achieve quality in an orderly fashion. The core benefit of test automation lies in identifying software bugs and fixing them as early and as close as possible to the developer’s coding environment. This helps subdue the damaging effects of late defect detection and also keeps the feedback cycle going between the engineering and product teams. Even though the upfront investment in automated tests might seem large, analysis has shown that, over time, it pays for itself. Test automation enables teams to deliver new features quickly as well as with superior quality. Figure 1.2 shows how various components of a software application can be interdependent, and continuously testing each of them asserts their behavior. It is not only necessary to validate these components as a standalone but also as an integrated system:
Figure 1.2 – Continuous testing
Imagine a simple loan application flow created by a combination of three Application Programming Interfaces (APIs). The individual APIs, in this case, could be as follows:
Each of the APIs has to be tested in an isolated manner at first to validate the accuracy of the business functionality. Later when the APIs are integrated, the business workflow should be tested to confirm the system behavior. Here, a test automation suite can be built to automate both the test cases for individual APIs and the whole system behavior. Also, when the applicant creation API is reused in another application, the automation suite can be reused, thus enabling reusability and portability in testing. A similar implementation can be done for user interface components, too.
Test automation is a very collaborative activity involving the commitment of business analysts, software engineers, and quality engineers/software development engineer in test (SDET). It unburdens the whole team from the overwhelming effects of repetitive manual tests, thus enabling the achievement of quality at speed. While peer code reviews and internal design reviews act as supplemental activities to identify defects early, test automation places the team in a fairly good place to start testing the product with the end users. It is a common misconception that automated tests undercut human interaction with the system under test. While it might be true that the tester does not interact with the system as often as they would in manual testing, the very activity of developing and maintaining the automated tests brings together the whole team by commenting on the test code and design. Automated tests open a new way of communication within the team to improve the quality of the system and the prevention of bugs.
Having looked at the basics of test automation, let’s learn some important considerations in the Agile world.
Having the right selection of automated tests at the right spots in the deployment makes a ton of difference in the quality of the delivered software. In an Agile environment, there has to be a constant discovery of the right tests for the current iteration, and the ratio between the manual and automated tests needs to be tweaked as and when necessary. Since the focus is on delivering a feature as quickly as possible to the customer, it is important that developers, testers, and the product manager are aware of what is being built, tested, and shipped. Collaboration becomes crucial and is the primary driver of the success of test automation in the Agile environment. Some of the important considerations for test automation in an Agile environment are as follows:
As we have seen so far, test automation is an intricate pursuit. Like all other complex things in the software world, test automation comes with its own set of challenges. Let’s examine some of them in the next section.
There is never enough test automation, and this is a constant challenge faced by test engineers. Test engineers are always under time constraints to finish the manual or automated tests and to get the completed feature out of the door. Just as with manual testing, there are a variety of challenges that test engineers face on a daily basis. Some of them are listed as follows:
Creating and maintaining the test automation infrastructure is a separate project by itself. Every bit of detail should be thought about and discussed in detail with the concerned stakeholders to address these challenges.
One of the chief purposes of test automation is finding regression bugs. Usually, regression means the quality gets worse after a change has been introduced to an already tested product. Well-written automated tests are tremendously helpful in identifying regression bugs. But an important thing to remember about regression issues is that even though more automated tests help find them, the root cause of regression issues still has to be addressed by the management team. Test engineers can provide awareness about the inherent regression issues through automated test results. It is important to educate the team about fixing the underlying lapses. Some of the most common slippages that cause regression bugs include the following:
Test engineers can be a key component to break the pattern of regression issues and steer the quality of the product in the right direction by keeping everyone on the team informed. In the next section, let’s survey some of the top metrics used in the test automation world.
Why and what to measure in test automation is a constant question lingering in the minds of the testing team. Test engineers are curious to know how effective their scripts are and how they are performing across different conditions. On the other hand, the management team will be interested to know the ROI on the investment made in test automation rather than just hoping that they deliver value in the long run. Let’s look at some key metrics that can help in gauging the value of test automation that is already in place:
Test Automation Effectiveness = (# of defects found by automation/Total number of defects found) * 100
Test Automation Coverage = (# of test cases automated/Total number of test cases) * 100
Test Automation Stability = (# of test failures/Total number of test runs) * 100
Test Automation Success Rate = (# of test cases passed/Total number of test cases run) * 100
Equivalent Manual test Effort = Effort to test a specific feature * Number of times a feature is tested in a test cycle
Test Automation Execution Time = Test Automation End Time - Test Automation Start Time
An important note regarding test automation metrics is that these numbers, by themselves, do not produce much value. A combination of these metrics properly assessed in the context of project delivery provides valuable insights to the product and engineering teams.
So far, we have seen how testing and test automation blend well with every activity in the software development life cycle. It is evident that test automation is a multidimensional undertaking, and the people performing test automation have to meet the needs of this demanding role. In the next section, let’s survey a couple of important roles in the quality engineering space.
There are three main roles in the testing world without involving people management. Different companies might use or call these roles in different ways. Quality-related roles also vary depending on the size and structure of the organization. For example, smaller companies might combine all of these into a single role and call it SDET. The main differences stem from their technical expertise and overall experience in the software and quality engineering domains:
The traditional manual tester role is not as prevalent as it used to be, as every role in quality is expected to perform some level of test automation in the Agile world. So, here, we will be focusing only on the latter two. This does not mean that manual testing is not done anymore. The responsibilities of a manual tester are shared by test engineers, SDETs, business analysts, and product owners/managers.
There are organizations where software engineers/developers perform their own testing. The engineer developing the feature is responsible for performing the required testing and delivering a high-quality product. Often, small- to mid-sized companies use this approach when they are still not able to build a separate quality organization or hire test engineers. An important downside of this is that there is a lot of context-switching that needs to be done by the developers. Also, consistently maintaining the test coverage at high levels becomes a challenge. A single developer will be focused on providing good coverage for their own feature, and it is very easy to miss the integration or system-level tests needed for a business workflow. Some developers, plain and simple, do not like to perform tests other than unit tests due to the tedious and repetitive nature of testing. It is also hard to keep track of the various details involved in testing different parts of an application.
Next, we will look at the most common testing roles that are prevalent in the market.
Test automation engineers are the core members of the quality organization within the software engineering teams. They can either be embedded into the engineering team or be part of a team of quality engineers reporting to a quality manager. The main responsibilities of a test automation engineer include the following:
To sum it up, on a daily basis, test engineers have to partner with most of the members of the team and stay on top of the user stories. They take ownership of the product quality of the team they are embedded into. Next, let’s see how an SDET is different from a test automation engineer.
An ideal candidate for the SDET position exhibits sound technical skills and a deep expertise in testing methodologies. For all practical purposes, technically, an SDET is as good as a software engineer with extensive knowledge of the quality engineering space. An SDET will be involved throughout the development life cycle from unit test creation to production release validation and always strives to enhance the productivity of both the software engineer and quality engineer in the team.
The main responsibilities of an SDET include the following:
Table 1.2 highlights the differences between a test automation engineer and an SDET:
Test automation engineer |
SDET |
Creates and executes automated and manual tests |
Creates and maintains the test automation framework |
Collaborates with the product and implementation teams |
Collaborates with software engineers and DevOps teams |
Highly skilled in programming with testing skills |
Experts in testing either manually or by automation |
Develops test automation tools |
Uses test automation tools |
Table 1.2 – Test automation versus SDET
Even though their roles demand different responsibilities, both the quality engineers and SDET are equally accountable for the release of a bug-free product to the customer. Both should be in the meetings with product stakeholders to make a final decision on every feature release. A quality engineer is good at test case creation, while an SDET specializes in choosing the right ones to automate in the best possible way. There are times when quality engineers and SDETs have to work in tandem to keep the upper management informed and educated about the capabilities of test automation and the effort it takes to achieve the ROI. Also, it is important to note that the relationship between software engineers and quality engineers/SDETs is of utmost importance to the success of any test automation work. It is vital to get continuous feedback from software engineers on the test automation code and design. Software engineers should also be educated, when necessary, about the various benefits derived from test automation
In the next section, let’s get ourselves familiarized with some commonly used definitions in the world of testing and test automation.
In this section, we will look at some of the most commonly used terms in the quality engineering space. Since quality engineering is part of the software engineering practice, you would notice quite a few familiar terms if you’re a software engineer:
We have equipped ourselves with the knowledge of a wide array of terms used in the quality industry. Now, let’s quickly summarize what we have seen in this chapter.
In this chapter, we looked at the importance of testing and test automation. We dealt with some practical guidance on how test automation interfaces with different teams and all the members of the engineering team. Additionally, we looked at the roles in the test automation world and their specifics and similarities. We understood how test automation helps achieve quality at every stage of the product development life cycle. Finally, we looked at the different terminologies used in test automation.
In the next chapter, we will examine what a thorough test automation strategy entails and how to define one. Additionally, we will review a few common test automation design patterns.
Take a look at the following questions to test your knowledge of the concepts learned in this chapter: