Ensuring internal and external quality validation

As you go through development, your internal quality assurance team develops a testing plan and tests the product. As the product manager, you should read and approve the plan to make sure quality assurance is testing the most common customer scenarios. Make sure that the plan includes a rigorous variety of common scenarios and setups that the target customer is likely to have in their environment. For example, for a web software product, make sure your internal testing team includes a variety of operating systems and browsers in their testing that are representative of what you customers use.

Rigorous internal testing will find many of the problems with your product. So why test externally at all? Because even with the best possible internal quality assurance engineers and testing, circumstances arise where real-world customers use the product in a different way that internal testing can’t anticipate.

Figure 13-1 helps you determine how big a need for a beta program your product has by asking you to evaluate the potential risk any post-release bugs pose to the customer against the difficulty of fixing them. For some products, such as consumer web software where the risk to the customer is low, you may want to run an open beta program for anyone. Google popularized this approach years ago as a way to release products without the company having to be responsible for bugs, problems, or things that didn’t work as advertised. In fact, many Google applications have never left beta.

For an application where the risk to the customer is low or for products that can easily be changed and updated rapidly after customers begin using them, the strategy of running a large open beta for anyone can be viable. For products that are mission-critical for customers or for products that can’t easily be changed — such as hardware, physical products, financial services software, or other applications that deal with sensitive data — it may not be a viable approach.

For all types of products, the risk you run in not holding an adequate beta program is that you may damage the brand reputation with customers and the market if the product isn’t solid. Even if it’s a web-based application, customers who have a negative experience may never be willing to use it again. Too many companies have put out products too early without getting adequate beta validation, and they simply could never recover.

Creating a beta plan

The beta plan is critical in that it helps formulate plans well in advance and ensure the corresponding beta program based on the plan will adequately achieve the desired results. The Product Management LifeCycle Toolkit (included as a free download with this book; see the Introduction, page 4, for details) includes a beta plan template you can use in creating your plan.

Beta programs take a lot of time and effort to execute in order to be done effectively. Table 13-1 suggests a timeline for each part of your beta program. Plan on 9 to 12 weeks from when you start planning the program, and make sure you have an employee or contractor dedicated at least half-time to running the program. Planning, recruiting customers, gathering feedback and suggestions, running an exit survey, and tallying the results to determine whether the product is ready to be released are all time-consuming yet critical activities that need to be covered.

Dodging typical beta testing mistakes

The most common mistakes made in the beta program are as follows:

• Not building enough time into the schedule to do adequate beta testing with real-world customers: Development almost always takes far longer than expected, and the company is anxious to release the product on time. As a result, the time set aside for beta testing is often reduced dramatically at the very end, resulting in an ineffective (sometimes nonexistent) beta program.
• Choosing beta testing customers that don’t represent the actual customers and personas: Friends and family often beta test. Though these people can provide some good feedback, don’t assume that they’re representative of how your actual customers will use the product or of the kind of environment customers will use it in on a day-to-day basis.
• Running a beta program without predefined goals and metrics: If you don’t establish what the goals and metrics are prior to running the program, you won’t be able to determine whether the product is actually ready to release. Having some concrete metrics in place can help make your decision to cut corners at the very end of development a far easier decision.
• Underestimating how difficult recruiting participants for the program will be: Many times, companies incorrectly believe finding customers who are willing to spend time beta testing the product will be easy. In real life, finding people who are dedicated, and willing to spend the time and effort required to provide useful feedback is often far more difficult.

Setting appropriate goals

Make sure you define your goals upfront and early. What are you really trying to accomplish? Do you want a broad cross-section of customers to spend extensive amounts of time using the product to prove that it’s ready to ship? Or is your existing quality assurance (QA) work extensive, and you just need a few customers to validate readiness with some hands-on customer usage? Do you want to gather early feedback for the next version? Or do you just want to find a group of customers willing to talk to the press or provide you with quotes or testimonials?

Making your goals concrete

The earlier you set your goals and the more concrete they are, the better. When you don’t define success upfront, creating your overall plan and knowing that you have reached your goals are difficult. In addition, if the testing shows that the product isn’t ready to ship, you need concrete goals that allow you to push back on the team and delay launch.

For example, your goal may be that a certain number of customers must have installed the product, used it for N days (or N hours or number of times) successfully, and found no major crashing bugs. One goal should be that you’ll survey the beta testers at the end of the program and find that a certain percentage of them (for example, 90 percent) indicate they believe the product is ready to ship. Flip to the later section “Exit surveys” for details on the survey part of the process.

Other goals may include finding three to five customers that are willing to give you quotes and talk to the press if reporters ask for references. A real bonus is if a customer is willing to create a blog post for you or recommend the product on your website or in social media. You may want to set a goal that the number of bug reports must show a decreasing trend to an acceptable level before the product can be declared ready to ship. Obviously, you can use a number of metrics.

Recruiting participants

Your success at recruiting customers depends on a number of factors, including whether it is a brand new and unknown product or an upgrade from a previous version, how well known and prominent your company is, and how popular your products are. For example, Apple could probably find iPhone beta testers pretty easily. Other factors include how easy finding your target customers is and how much effort and time you’re asking them to put in.

If you have an enterprise software application from a new, completely unknown start-up company that will require the beta customers to go through a lengthy installation (or worse yet, installation may affect their other mission-critical systems), you may have a difficult time recruiting. On the other hand, if you have a consumer application that takes little time to install or try out and provides immediate benefits, you may find recruitment easier.

Depending on all these factors, you need to gauge what kind of recruiting program you need to do and how strong the incentives need to be (see the section “Incentives” later in this chapter). You may need to make personal phone calls and visits in order to convince customers to participate. Or you may be able to use email or even a form on your website.

Sources for recruiting include the following:

• Current customers
• Prospective customers that didn’t purchase before
• Venture capitalists and investors who can refer you to representative customers
• Your personal network (but be careful using friends and family; see the earlier section “Dodging typical beta testing mistakes”)
• Your sales force and leads
• Advertisements (online, in the local newspaper, and so on)