Considering Skill Testing Styles

Fortunately, there are several useful tools in the Alexa development ecosystem for testing skills. In fact, there are enough choices that it can be difficult to know which testing tool to use. To help you understand which testing tools are the best fit for a particular circumstance, consider that there are three styles of testing for Alexa skills:

• Vocal/Manual
• Non-Vocal/Semi-automated
• Non-Vocal/Fully automated

Vocal testing is almost completely manual, requiring that you speak to an Alexa-enabled device or simulator and listen to the response to know if your skill is behaving as expected. One of the key benefits of vocal testing is that it is virtually the same as what your end-user will experience when they use your skill. Also, unlike the other non-vocal testing styles, you will be able to verify how your skill sounds. This is incredibly important if your skill includes sound effects, music, or uses the Speech Synthesis Markup Language (SSML) to alter Alexa’s voice or tone. (We’ll look at SSML in Chapter 6, ​Embellishing Response Speech​).

While vocal testing has its benefits, it also has a few limitations. First, it requires that the skill be deployed before you can test it. If changes are required between test runs, you must wait for the skill to be redeployed, which gets in the way of quick feedback on your changes. With vocal testing, it is also not easy to reliably repeat a vocal test and ensure that you said and heard the same things each time. Finally, vocal testing is difficult, and perhaps even rude, to conduct in a crowded place such as an office, airplane, or StarBucks.

Non-vocal testing, as its name suggests, does not require speaking or listening to a skill, thereby sparing those around you from hearing your conversation with Alexa. Instead, non-vocal testing involves submitting your questions and commands in textual form, either by manually typing them in or as part of a script. When submitted as a script, the tests can be reliably repeated and run as many times as necessary.

Semi-automated non-vocal testing still relies on you to verify the results manually, albeit by reading results instead of listening to them. It also requires that the skill be deployed before you can test it, and thus the feedback loop can still be lengthy. In spite of these limitations, semi-automated non-vocal testing can be nearly as flexible as vocal testing when it comes to trying different things, allowing you to mix repeatable tests with ad-hoc tests if you like.

While vocal testing of a skill brings us as close to the user experience as possible and semi-automated testing is useful for ad hoc testing and debugging of Alexa skills, both techniques are unfortunately dependent on human participation. And humans are notoriously unreliable when it comes to doing repeated tasks such as software testing.

Computers, on the other hand, excel at automatically performing repeated tasks. Therefore, if we are looking for a way to reliably and frequently verify that our skill is behaving as we expect it to behave, then we should consider writing fully automated tests.

Fully automated non-vocal testing is repeatable, and verification of results is scripted as part of the test itself, requiring no visual or audible verification. This makes it a perfect fit for a CI/CD pipeline to build and deploy your skill automatically when changes are made. In this style of skill testing, the skill does not need to be deployed. This means that the feedback loop is tight, enabling you to develop faster and with confidence that your changes are good.

The biggest downside of fully automated testing is that it is far removed from the actual user experience, and as such there’s no way to experience the skill as your users will experience it. But being able to reliably repeat tests and verify results makes it incredibly powerful.

Because vocal testing requires no special techniques or tooling, aside from an Alexa device to talk to, you’re welcome to establish your own strategy for vocal testing. In this chapter, we’re going to focus on techniques for non-vocal testing. We’ll start by looking at how to write semi-automated tests that let us interact with our skill by typing our queries and reading responses.