Gathering Usage Metrics

Now that your skill is live, you may already be thinking about changes you want to make for a new release. You might have an all new feature in mind. Or you may have some ideas on improving the interaction model to make it handle user requests more accurately. And, it’s possible you’ve found some bugs that need to be sorted out in the next version.

Before diving into the code to address these concerns, it can be very informative to review some of the metrics around your skill’s usage. Amazon automatically tracks several helpful metrics about a skill, requiring no additional coding or configuration. You can use these metrics to learn how many customers are using your skill and how they are using it. With these insights, you can better plan how to prioritize changes to your skill.

Skill metrics are available both from the developer console and via the ASK CLI. Let’s have a look at the how metrics are presented in a graphical form in the developer console first.

Viewing Analytics in the Developer Console

If you visit the developer console after your skill has been published, you’ll notice that there are two entries for the skill in the list of skills. The first screenshot shows what you might see.

The first entry represents the published skill, while the second entry represents your skill in the development stage. As you continue to develop your skill for future versions, it’s the second entry that you’ll be working with. The first entry, however, is where you’ll go to view the analytic data for your skill. You can either click on the “Analytics” link in the Actions column or click the skill name and then click on the “Analytics” tab. The second screenshot shows the analytics summary page.

As shown here, the summary page includes some basic information such as the number of sessions, unique customers, and utterances handled by our skill over the past week. This screenshot was taken only a couple of weeks after the skill was submitted for certification, so the only data available was created by the certification process and isn’t sufficient for some metrics to be shown. For example, the user enablements and account linking completion rate metrics require a significant amount of data before they’re shown.

As you dig into the analytics pages, you’ll find more information, including historical metrics presented as graphs. For example, clicking the “Sessions” item in the left-hand menu will show several graphs pertaining to session count, as shown in this screenshot:

Feel free to explore the various graphs and metrics offered under the “Analytics” tab on your own. As you do, be sure to have a look at the “Interaction Path” graph. This graph shows the various paths users have taken through the skill, going from one intent to the next until their session ends. An example of what this graph looks like is shown in the screenshot.

The interaction path graph requires a lot of data before it can be displayed; this screenshot was taken from the analytics page of a skill that was published some time ago. But once it is available, the interaction graph gives a great visualization of how your skill’s users are using the skill. You can use this information to improve the skill. For example, if you find that a lot of users are following lengthy paths through several intents to get to a piece of information, you might optimize your skill to make the same information available with a single intent.

Accessing Skill Metrics with the ASK CLI

While the graphs provided in the “Analytics” tab are generally useful and interesting, you might find a need to crunch the numbers yourself or create your own custom visualizations of the metrics. If so, then the ask smapi get-skill-metrics command is what you’ll need.

To give the get-skill-metrics a try, let’s use it to find the total sessions per day for a specific week in June 2020:

The get-skill-metrics command requires several parameters, including:

• skill-id—The ID of the skill to fetch metrics for

• start-time and end-time—The range of time for which metrics should be fetched

• period—The granularity of the metrics. Can be “SINGLE” (a single count for the entire range), “P1D” (per day), “PT1H” (hourly), or “PT15M” (every 15 minutes)

• metric—The specific metric you are interested in. Can be “uniqueCustomers”, “totalEnablements”, “totalUtterances”, “successfulUtterances”, “failedUtterances”, “totalSessions”, “successfulSessions”, “incompleteSessions”, “userEndedSessions”, or “skillEndedSessions”

• stage—The stage of the skill, either “live” or “development”. This defaults to “development”, but in this case we’re more interested in the metrics for our “live” skill.

• skill-type—The type of skill. Often “custom”, but can also be “smartHome” or “flashBriefing”

As you can see, the response includes timestamps and values properties. The timestamps property is an array of timestamps for each point that data is recorded. Each entry in timestamps corresponds to an entry in values which is the value of the metric at that timestamp. In this case, there are seven dates with one value for each date falling within the specified date range.