Chapter 13. Agile QA and Production

Part of the role definition for quality assurance and production is the ability to communicate with all disciplines. Since fostering a common language across all disciplines is a principle for agile game teams, QA and production have an advantage when agile is adopted.

At the same time, agile practices change the role of QA and production the most. Fortunately, the change is for the better. This chapter addresses those changes and how the QA and production roles fit into agile teams and organizations.

Agile QA

Every year I’m invited to my sons’ school for parent career day. It’s always fun. To the class, “video game developer” is the coolest career imaginable. I could be standing next to the “astronaut/secret agent” dad wearing his space suit, and the kids would climb over him to ask me questions.

An inevitable question is, “How can I be a game tester?” They dream that the role of a tester is eight hours a day of the same fun they experience playing games. What could be better than that? Unfortunately, I am forced to destroy this notion. I describe the long hours with games that crash. I describe how testers are often very frustrated because they have an understanding of what the game is and have very little influence over how it is developed. I don’t want to discourage future generations of testers; I just don’t want to give the impression that there is this wonderful job waiting for them because they are “good at playing games.”

As you may guess, the QA role attracts people who are passionate about games. The sheer number of people applying for this role gives the industry the ability to choose from among the best. We need to leverage their passion and experience far more.

In this section, we’ll learn how to do that. We’ll explore the role of QA on a game development project, and we’ll explore how it changes on an agile team.

The Problem with QA

Traditionally, quality assurance is largely performed at the end of the project by a dedicated QA team. Figure 13.1 shows the pattern where most bugs are discovered on such a project after alpha, when all features are considered complete but not debugged.

Figure 13.1. Bug discovery rate on a traditional project

The reason for this is simple; testing is performed on a potentially shippable game. Since the typical game project does not achieve this state until post-production, most testing has to be compressed between the alpha and ship dates. As a result, armies of testers are hired in hopes of achieving a ship date with a working game. It is impossible to hire and properly train a small army quickly, so testers are minimally trained for each game.

This type of quality assurance doesn’t always assure the highest quality. By alpha, the important decisions about the game have been made, and the job of QA is to find and report minor defects. Most major defects are rooted too deep in the design or architecture of the game to be properly addressed. For example, if level pacing isn’t fun, it may be too late to correct it. This compromises the quality of the final game.

The other problem with traditional testing practices is that quality becomes the responsibility of remote¹ QA and not the developers. Studio cultures encourage this by basing employee performance evaluations on the pace of feature implementation or asset creation. Bugs that do not stop progress are considered part of making the game and are tolerated before alpha. Adding QA practices during this time slows feature and asset creation in the short-term, so it is deferred.

¹ Such as a publisher’s QA department in another location or a QA service

Agile Testing Is Not a Phase

The approach to QA on an agile project is different from that of a traditional project in a number of ways:

• Testing takes place on an agile project throughout its life cycle: This is a major benefit of creating a potentially shippable version of the game every sprint. Defects are not deferred or tolerated.

• Testing is white-box testing: Testing focuses on the internal components of the game rather than the game as a whole. This requires more skilled “product analysts” who can specialize in areas such as code reviews and test automation.

• QA is part of every role: Each developer performs QA in their daily work. QA doesn’t eliminate this need but catches problems that slip through and helps improve team practices so that similar problems are more likely to be caught by the developer in the future.

Although large testing groups are still built up in post-production, an agile project maintains a core QA group throughout the project. The goal is to catch most problems as they occur, well before alpha, as shown in Figure 13.2.

Figure 13.2. Bug discovery rate on an agile project

The Role of QA on an Agile Game Team

Compared to traditional approaches, agile projects extract more value from testing. Much of this value comes from each team’s goal of delivering a playable and potentially shippable game every sprint. The QA role shifts to take advantage of this:

• Testers are members of teams, rather than gathered in separate QA pools. This increases the speed of feedback on defects to the developers.

• Stories require QA action for approval before they are considered done by the development team.² This creates a true pace of development, which should include fixing defects and reduces the cost and added risk of fixing them at a later date.

² The product owner has the final say, however.

• QA’s voice is heard throughout development, not just at the end. This allows their valuable feedback to influence the game.

QA’s role on an agile team grows beyond its traditional role. It starts with QA’s participation in sprint planning. As each story is discussed, QA needs to understand how that story will be approved, before tasks are estimated. This usually takes the form of how the story and the conditions of satisfaction (CoS) are written.

Consider, for example, the following story:

As a player, when I hit the jump button, my character will jump.

The team discusses this story and may ask the following questions:

• Will the character have jump animation? Will it need smooth transitions?

• Can the character jump to a higher level?

• Can the character jump from a moving platform?

• Can the character jump while walking or running?

These questions help the team understand the tasks necessary to implement and test the story to ensure it is done. For example, the answers to the previous questions may define the following CoS:

• The character will have a simple stand-in animation that will not be blended.

• The character will jump up to a ledge or down to a lower level.

• The character is not expected to jump correctly from a moving platform.

• The character can jump from any starting motion, but its momentum will be ignored. It can jump only a fixed distance and height.

Using these CoS, both QA and the developers verify that the story is complete before the end of the sprint.

QA should also locate problems that are not part of specific CoS but are part of development. For example, if the test level is missing some physical geometry that enables the player to “fall through the world,” QA should identify it as an impediment.

A tester should help to ensure that the team is not impacting other teams. This includes the following:

• Performing regression tests on areas the team is working on that could break other areas of the game: For example, if the team is working on character control for jumping, keep an eye on other character motions to ensure that some side effect of the team’s changes is not breaking them. Another example would be if every time a player dies, their character assumes the jump pose, then the team working on jumping has probably broken something!

• Testing tools and pipeline changes that a support team wants to release to the content creators: This includes testing beta versions of a tool with the content creators to verify changes have not broken a workflow.

• Finding ways to improve testing: For example, if bad textures are frequently crashing the game, QA should raise a request to have the texture exporters improved to catch the problem.

Some of QA’s time is spent verifying gameplay and offering advice on areas of improvement. This requires an ongoing dialogue with members of the team. Since QA is a voice for the consumer, they need to keep a “consumer eye” on the game. They should point out any issue that the consumer will notice (such as progression stoppers).

QA will be challenged in this new role. They are required to communicate with every discipline, understand the tools, and stand up and be heard. QA has traditionally been a “gateway role” for other roles in game development, so it’s appropriate to develop these skills to grow their career in game development.

QA, Embedded or in Pools?

To what degree should testers be spread across teams or located in a QA pool that tests independently? There is no simple answer beyond “It depends.” It depends on what the team and project need in order to ensure that quality is built into everything they are working on. It depends on the skill level of the testers and how much white-box testing they can accomplish. It depends on the testing tools, harnesses, and telemetry that are available.

When a game is in early development, the depth of gameplay and limited assets don’t need as much separate testing; teams might share testers. As the project progresses, there is more to test. The game may need to run on several platforms. The growth of complexity requires not only more testing of the added features but more regression testing to ensure that previously added features have not been inadvertently broken. QA pools can even out this workflow, helping team testers in times of need. As a game approaches its ship date, QA may be concentrated in a single pool or in separate pools dedicated to each platform.

When a tester is embedded within a single team, they take on a more expanded role. They can assist the team in addressing impediments and completing tasks. For example, if an artist needs to test an asset on the PS3, an embedded tester can help them by creating a PS3 build, launching it, and calling the artist over when the asset is visible. Embedded testing encourages a tester to build their white-box testing skills and knowledge to overlap their work with the other developers on the team.

How Many Testers per Team?

A frequent question raised is, “How many testers does a Scrum team need, and when should they be added?” It’s the same as asking how many animators or audio designers a team needs; the answer is “as many as you need!”

The number of testers depends on the testing needs of the team. This hinges on the following:

• The definition of done: As the definition of done for sprints approaches the final shippable state, more testing is required. For example, if a definition of done requires the game to be completely playable from start to finish without crashes on any platform, more testers are required than in early development when the game has to run on a development PC for a few minutes at a time.

• Team testing practices: If the team uses practices that support higher quality, such as TDD or thorough export testing, there will be fewer problems that QA is needed to find.

• Test automation: It depends on the coverage level of automated test utilities, ranging from simple smoke tests to complex scripting, that allow the game to “play itself” and offload manual testing.

• Playability testing needs: The game needs hands-on, in-depth testing from a player’s perspective. QA needs to evaluate usability, pacing, and difficulty levels and provide feedback to the team.

Typically, one tester per team is enough for most of the project, but ultimately the team’s needs are driven by sprint results. As the stakeholders raise the definition of done, it may put pressure on the team to recruit more testers to help them to meet their goals.

Using a Bug Database

As discussed in Chapter 7, “Video Game Project Planning,” a debugging stage after the alpha date is unavoidable for many games. For these projects, the QA staff is expanded, and the entire game is tested from beginning to end.

It’s dangerous for a project to rely too much on post-production testing and allow significant defects to remain undetected throughout production. When such defects are uncovered after alpha, they can invalidate a great deal of work. For example, discovering that the disc budget has been exceeded by 50% in post-production is very bad.

Ideally, post-production efforts should focus on tuning and polishing tasks that are identified and prioritized daily. Even without major flaws, tracking polishing, tuning, and minor bug-fix tasks can overwhelm a sprint task-board approach.

It’s important that there be only one product backlog for the team. It’s important for the product owner to prioritize a single list of work for the team to draw from every sprint. Before alpha, any bugs that are addressed must either come from the team during the sprint or come from the product backlog. If a bug database is maintained prior to alpha, then any bugs that are to be worked on are moved from there and placed on the product backlog to be prioritized by the product owner.

At alpha, all unimplemented stories on the product backlog are cleared off (they may become part of the product backlog for the next version of the game). From here out, only bug fixing and tuning work can be placed on the product backlog. The product backlog might even be replaced by the bug database.

The product owner manages the priority of bugs with QA through frequent triage sessions. As with hardening sprints, described in Chapter 6, “Agile Planning,” the team draws a set of tasks to work on daily, without a specific sprint goal. The new goal is to burn down all the high-priority bug fixes and tuning work.

Play-Testing

A major benefit of producing a potentially shippable version of the game every sprint or release is that a game is tested early by potential consumers. Play-testing is a practice where consumers are lured in to play a game in development—often with the offer of free pizza—and provide useful feedback about the gameplay experience.

Play-testing can be as informal as a conversation. It can also take a more scientific approach by recording progress metrics or the answers to carefully designed surveys.

Play-testing has substantial, but limited, benefits:

• It reinforces a definition of done: Developers, who see a game every day, overlook flaws or shortcomings. Play-testing is often a shock when they observe how these problems impact real consumers. For example, that missing piece of collision geometry is always found by play-testers!

• It won’t produce any breakthrough ideas: Don’t expect the play-testers to provide original ideas. That’s the developer’s job. At best, they provide feedback that can improve backlog prioritization.

• It teaches about usability and challenge: Ever see a player simply run past a painstakingly scripted boss? It’s a sobering thing to see. It’s also very frustrating to see a player walk against a wall for two minutes until you realize that they are trying to make sense of a HUD map that is too vague!

Play-testing is often organized and run by QA or usability specialists. These duties include the following:

• Recruiting play-testers: Local universities and game retail shops are teeming with good play-test candidates. Be sure to enlist people who represent the full scale of demographics and skill levels. For example, a hard-core gamer won’t provide the same feedback about the tutorial level as a casual gamer. Maintain a database of people, and invite the more valuable play-testers back.

• Organizing and running the session: A poorly organized and run session elicits inferior feedback from play-testers. Don’t waste their time.

• Including the developers: It’s very beneficial for the developers to see players interacting with their game in ways they did not anticipate. It can lead to improved interfaces and usability (see the sidebar “Damage Meter”).

• Letting play-testers meet the other developers: Many testers are curious about game development and relish the opportunity to speak with the rest of the team at the end of the session. Encourage this. The conversations can be very fruitful!

• Publishing the results: QA’s observations of how well the game was received provide many insights. However, care must be taken not to “overinterpret” the results to include subjective bias from QA. A simple compilation of the answers helps the team understand the results. Leave the conclusions to the readers or discuss them together.

Maintaining the playability of the game allows everyone to value the working game over the comprehensive design.

The Future of QA

As the game industry continues to improve agile practices, the role of QA continues to transform. As testing becomes more white-box driven, testers require more knowledge about development disciplines, and testing specialties will grow. For example, test engineers in many industries focus on the quality of the code being produced by programmers. These specialists run code-quality scanner tools³ and measure unit test coverage to ensure that coding practices are being held to a high standard. Is this a testing role or programming role? It’s a bit of both, and it represents a typical evolution and blurring of roles in an agile environment.

³ Such as PC-Lint

The rising importance, skill level, and value of testers on an agile game project will improve the role and make it a truly desirable career.

Agile Production

Producers are the most open to adopting agile but usually with some trepidation. They are the first to foresee its benefits, but they are also concerned about the implications to their role on an agile game project. The main concern is that as teams become self-organizing and self-managing, then there will be no need for producers. In reality, producers are often the engines of ongoing agile adoption. As with the discipline lead roles, the production role is divested of many of the mundane tasks with agile, such as tracking hourly estimates and solving the myriads of small problems that cross-discipline teams handle themselves. Instead, an agile producer expands their focus on the big picture while helping teams achieve continual improvement in how they create games.

This section identifies the changes in responsibility for the traditional producer on a Scrum project and how they can assume the ScrumMaster, product owner, or product owner support role.

The Role of a Producer on an Agile Project

Producers are often seen as the person who makes sure everything gets done. This is not the case on an Scrum project. As Scrum teams take responsibility for what is done every sprint, they relieve the producer of some daily project management tasks:

• Building and maintaining detailed schedules: Scrum teams create and estimate their own tasks.

• Tracking daily tasks for each member of the development staff: This is managed in the daily scrums by the team.

• Managing dependencies: Cross-discipline teams manage dependencies daily and in sprint planning by selecting goals and team members who avoid external dependencies.

Instead, the producer should focus on the larger project management challenges:

• Tracking external dependencies: Will the outsourced cinematics be delivered on time?

• Publisher collaboration: Are the stakeholders playing the game if they can’t attend a review?

• Outsourcing support: What does the outsourced level art team need to complete their work?

• First-party (platform) support: When are we scheduled for acceptance testing, and what are the requirements?

• Risk management: What are the external schedule and resource risks for obtaining licensee approval or assets?

• Critical chain management: What are the resource schedule needs for the production assets?

Most of these responsibilities are best described in a number of good game production books (see the “Additional Reading” section at the end of the chapter).

Producer as ScrumMaster

Many studios adopting agile struggle with identifying people to best fill the ScrumMaster role. It’s often assumed that the producer is natural for the role, and many times this is true. There are benefits and potential drawbacks with a producer taking on the ScrumMaster role.

A benefit seen with producers in the ScrumMaster role is the ability to communicate equally well with all the disciplines. This is essential for a cross-discipline team. This allows them to remain unbiased toward issues or unconflicted about what they need to do. For example, programmers who become ScrumMasters often see issues through a filter that says most problems are best solved with code. This leaves the other disciplines feeling less supported.

Another benefit is that producers usually have fewer tasks to commit to every sprint. This prevents them from having to choose between completing critical tasks and supporting the team. Depending on the team, the ScrumMaster role can take anywhere from 33% to 100% of a person’s time.

A common drawback with a producer taking the ScrumMaster role derives from their past duties on more preplanned projects. These duties required producers to lead individuals toward task completion. However, Scrum teams manage their own tasks to achieve a sprint goal as a team. The ScrumMaster does not interfere with the team by creating, estimating, assigning, and tracking tasks for them. Unfortunately, some producers assume, out of habit, that the ScrumMaster role continues to focus on tasks. This prevents teams from achieving all of the benefits associated with self-management.

Producer as Product Owner Support

On large game projects, the demands of a product owner’s time has led to hierarchies of product owners (see Chapter 9, “Faster Iterations”). Similarly, the long-term demands of production planning, resource requirements, marketing, licensing, and first-party hardware support have led product owners to seek production support.

As a result, it is common to pair a product owner with a senior producer. The product owner manages vision while the senior producer attends to many project management details. This support enables the product owner to make better decisions based on license, franchise, budget, and schedule limitations.

License and franchise details are usually managed by the product owner, but there are details that need to be handled on a day-by-day basis. For example, providing sample assets or seeking approval on the use of brands or likenesses can be very time-consuming.

The most important schedule and budget limitations are those that define the production requirements and dates. Pre-production creates cost and schedule debt that is paid off in production. The product owner needs to monitor this debt to ensure it doesn’t grow beyond the limits of schedule and cost defined by the stakeholders. For example, before defining the number of characters to produce, the cost for a single shippable character needs to be understood.

A product owner and senior producer make an effective pair to create the best game at the right cost and schedule.

Producer as Product Owner

Many successful games have been led by a senior producer who provides project management and vision for the game. As with lead designers who have strong project management skills, a senior producer with a strong vision of what the market wants is rare and invaluable.

When a studio has such a producer, the product owner role is ideal for them. The role leverages vision and project management strengths like no other (see Chapter 3, “Scrum”).

The Future of Production

As the industry becomes more agile, the producer role will include specialized and team roles.

Teams no longer need the ubiquitous “gopher” but will need the support of the following specialists:

• Outsourcing/insourcing

• Licensing

• Franchise management

• Production planning and support

• Technical production (including first-party communication)

Producers can successfully assume the ScrumMaster and product owner roles as well. Production will not disappear from agile studios but flourish.

Summary

Agile QA and production will continue to draw people who have these primary strengths:

• A deep passion for video games

• Great organizational skills

• Great communication skills

These strengths are in demand in every industry, and the game industry has to compete for them like any other. By leveraging them, agile organizations reap major benefits from QA and production, while testers and producers receive greater challenges in which to rise.

Additional Reading

Chandler, H. 2008. The Game Production Handbook, Second Edition. Sudbury, MA: Jones and Bartlett Publishers.

Crispen, L., and J. Gregory. 2009. Agile Testing: A Practical Guide for Testers and Agile Teams. Boston: Addison-Wesley.

Hight, J., and J. Novak. 2007. Game Development Essentials: Game Project Management. Clifton Park, NY: Delmar Cengage Learning.

Irish, D. 2005. The Game Producer’s Handbook. Boston: Course Technology PTR.

Isbister, K., and N. Schaffer. 2008. Game Usability. San Diego: Morgan Kaufmann.

Laramee, F. D. 2003. Secrets of the Game Business. Boston: Charles River Media.