Building an API.AI Agent

For our agent build, we will follow a simple step-by-step process (you’ll need a Google account to access API.AI, as the company was recently acquired by Google):

1. Open a web browser and visit http://api.ai. You should land on the page shown in Figure 5-11. Click GO TO CONSOLE at the upper-right section of the window.

   

   Figure 5-11. API.AI landing page

   You will then be directed to the login page, shown in Figure 5-12.

   

   Figure 5-12. API.AI login

2. Go ahead and log in with your Google account credentials. You should then be redirected to the page shown in Figure 5-13. From there, click the CREATE AGENT button.

   Note

   If you are prompted to authorize API.AI due to insufficient permissions, simply grant the required access to continue the process.

   

   Figure 5-13. API.AI landing page

3. You should now be presented with the Create Agent form (Figure 5-14). Give your agent a name and a brief description, and set your default time zone. We don’t need to provide sample data or create a Google project for our “Hello, World” example, so you can simply click Save to initialize our agent.

   

   Figure 5-14. API.AI’s Create Agent form

4. Once your agent is successfully created, you will be directed to the page shown in Figure 5-15. This is where we can create the intents and entities for our agent. Let’s build our HelloWorldIntent now. Click CREATE INTENT.

   

   Figure 5-15. API.AI Agent console

5. There are a lot of options on the “Create Intent” form (Figure 5-16), but we only need to focus on three tasks: naming our intent, defining “User says” utterances, and building a list of responses. First things first, let’s name our intent HelloWorldIntent. Next, enter the utterances we expect our users to say for our “Hello, World” example. For example, “Say ‘hello’ to the world” or “Say ‘Hello, World’”. As you can see, this is virtually identical to how we create utterances for Alexa in the Amazon Developer Portal. Next, let’s create a list of possible responses that can be returned whenever our HelloWorldIntent is triggered. This is one of the more powerful features of API.AI. As you’ll recall from Chapter 4, we had to create a lambda function to handle and build the responses for our Alexa skill. All of our skill logic was managed in code. However, with API.AI, we now have a code-free way to not only return a response but also to randomize the phrases returned. This makes our interactions more interesting and conversational, all without writing one line of code. As already mentioned, there are many other items available on the Create Intent form, but we won’t cover these here; however, we encourage you to experiment with each of the available options. For now though, just go ahead and save the work we did for the HelloWorldIntent.

   

   Figure 5-16. New Intent form

6. After saving your HelloWorldIntent, you will be brought back to the main agent console where you should now see your HelloWorldIntent listed (Figure 5-17). Before we begin building our UWP app, let’s take our agent for a test run. Type Say 'Hello' in the “Try it now” box and hit Enter. This will trigger your agent. You should then see one of your HelloWorldIntent responses listed under Default Response. You should also see the name of the intent your agent triggered based on the Say ‘Hello’ utterance. If everything worked as expected, congratulations! You successfully created a custom API.AI agent. A quick note on the two default intents, which are included in every agent. The Default Fallback Intent is triggered whenever your agent can’t find an intent for a given user utterance. API.AI provides a nice list of canned responses as part of the “fallback” intent, but you can modify the list as needed. The Default Welcome Intent is triggered by Welcome Events, typically fired by social media platform bots, although it can also be triggered via the API.AI API. Our “Hello, World” example won’t utilize the Welcome Intent, but we wanted to explain its usage.

   

   Figure 5-17. Agent test

7. There are just a few more things we need to do in preparation for our UWP app integration. First, click the COPY CURL link (Figure 5-18). This will provide a sample HTTPS call we can use to connect our UWP app to our API.AI agent.

   

   Figure 5-18. Test output

   The copied curl statement looks similar to this:

   	curl 'https://api.api.ai/api/query?v
   	=20150910&query=say%20hello&lang=
   	en&sessionId=4f1b46ac-1ef7-
   	43ad-a502-3310d6e1f094&timezone=2017-07-08T12:51:48-0400'
   	-H 'Authorization:Bearer
   	   092e4a9c2334fed9e4f50e7ebf56d2d'
   	

   We’ve highlighted the important bits. API.AI requires certain parameters for query calls. Specifically, we will need to provide the following query string parameters:

   Query

   The actual utterance spoken by our user

   SessionId

   A random string (we will use a GUID)

   Lang

   The language of the response (because we are working with one language in our example, we’ll pass “en” for English)

   V

   The version of API.AI API we wish to use

   Time zone is not required, and while Bearer token (Client Access Token) is mandatory for each request, it will be passed as part of the HTTP request header. We’ve reformatted the request to give us a base URL, which we will use in our UWP app (take note of this URL, you will need it later in the chapter):

   //our modified base URL for API.AI
   https://api.ai.ai/api/query?v=20150910&lang=en
   
   

   You will need your Client Access Token for our UWP app. To retrieve it, simply click the gear icon next to your agent name at the top-left of the main agent console (see Figure 5-19). Your Client Access Token will be listed under the API Keys section.

   

   Figure 5-19. Click on the gear icon to retrieve your access token

8. The second item we need to cover is the response object returned from API.AI. If you click the SHOW JSON button shown in Figure 5-18 it will reveal the response JSON returned from our HelloWorldIntent (Figure 5-20). The element we are most interested in is contained within the fulfillment object. The speech element contains one of the response phrases we entered in the HelloWorldIntent. This is the value our UWP app will need to parse from the JSON object. (There is also a speech element within the messages array. This is a text value supporting line breaks to support Facebook Messenger, Kik, Slack, and other messaging services. You can certainly use this value, but it’s not required.) You don’t need to take any action at the moment; this image is provided simply as a reference to the API.AI response object. Full details regarding the response object can be found at https://api.ai/docs/reference/agent/query#response.

   

   Figure 5-20. API.AI HelloWorldIntent response JSON

9. At this point, we’ve completed all of the API.AI requirements for our “Hello, World” example, but we want to quickly show you how easy it is to add Small Talk to your agent. Select Small Talk from the left menu and you will be presented with the screen shown in Figure 5-21. As you can see, API.AI provides a fairly comprehensive list of Smalltalk topics to choose from.

   

   Figure 5-21. API.AI Small Talk dashboard

10. Click the “About agent” topic and under “Who are you?” (Figure 5-22), and enter a list of random response phrases you want your agent to return for that question. When you are done, click Save. That’s it! Now your agent will respond with one of your provided phrases whenever it receives the “Who are you?” question. Obviously this greatly simplifies our agent’s ability to handle commonly encountered phrases. Pretty powerful stuff!

    

    Figure 5-22. API.AI Small Talk “About agent” configuration

OK, that’s a wrap on our API.AI effort. We were able to create a rich set of intents and responses all without writing one line of code. We were even able to add a bit of Small Talk handing to the mix. As mentioned earlier, API.AI is a very capable NLU/NLP engine, and we highly recommend it for any voice-based project. It’s especially handy for building codeless bots using API.AI’s pre-built integrations. Experiment with API.AI—you won’t be sorry you did.

Now, let’s go build our UWP app and tie it all together.

The Plan

The overall plan is to have fun creating a voice-activated UWP app. To help us achieve this goal, our app will leverage Microsoft’s Windows.Media namespace. Specifically, the Windows.Media.SpeechSynthesis and Windows.Media.SpeechRecognition classes.

The SpeechSynthesis class will be used as our audio playback engine, playing the responses we receive from API.AI. The SpeechRecognition class will be used to listen and capture our user’s voice requests (utterances). We will then forward the utterances to API.AI for processing and parse and play the received responses.

Let’s get started!

The Code

This section breaks the build into simple steps and explains the critical elements of the code. For convenience sake, we’ve made the code for this project available on GitHub. Clone this project locally. We will demonstrate how to create and configure a new UWP project, but you can leverage our existing code to get you up and running quickly. Here are the steps you’ll need to follow:

1. Go ahead and open VS and create a new project. Select Blank App (Universal Windows). Name your project and solution and click OK (Figure 5-23).

   

   Figure 5-23. New UWP app

2. When creating a UWP, you will be prompted to select a target and minimum version of Windows 10. You can leave all the defaults and select OK (Figure 5-24).

   

   Figure 5-24. UWP targets

3. Your project should look similar to Figure 5-25. We need to do a little configuration housekeeping so double-click the Package.appxmanifest file.

   

   Figure 5-25. A typical UWP project structure in Visual Studio

4. Select the Capabilities tab and then ensure that both Internet (Client) and Microphone are selected under the Capabilities list (Figure 5-26). This grants our app access to the microphone hardware.

   

   Figure 5-26. UWP package manifest settings

5. Now we need to set the Target platform from x86 to ARM (the Pi is an ARM-based processor). On the main toolbar, click the drop-down box adjacent to the build mode (debug/release) and select ARM. Next, click the arrow next to Local Machine and select Remote Machine (this will be our Pi), as shown in Figure 5-27.

   

   Figure 5-27. Set platform to ARM and select Remote Machine

6. Upon selecting Remote Machine, you should be prompted to enter a Target Device, Remote Machine, and Authentication Mode. Ensure the settings are configured as indicated in Figure 5-28. Be sure to enter you Pi’s IP address for the Remote Machine setting.

   Note

   If you are not immediately prompted with a pop-up box when selecting Remote Machine, you can make the adjustments by navigating to Debug→Properties and then selecting the Debug menu option (see Figure 5-28).

   

   Figure 5-28. Set remote machine to our IoT Core Pi IP address

7. Now that our app is configured correctly, let’s set it aside for a moment and open the solution you cloned from GitHub. This will give us an opportunity to review the code in detail. When we finish the remaining steps, you can either copy and paste our code into your new project, or simply run our cloned project.

   Note

   If you choose to copy and paste our code into your project, remember to replace the Client Access Token (APIKEY) to match your API.AI Token. If you choose to simply run our cloned project, you will need to update the Client Access Token as well the Remote Machine IP Address.

8. Once the cloned solution is open, expand the MainPage.xaml item and then double-click the MainPage.xaml.cs file to reveal the source. We’ll start at the very top. First, we initialize an instance variable SpeechRecognizer object named recognizer, which will be used to start a continuous recognition session. The continuous session will listen for an invocation word (similar to how Amazon devices listen for “Alexa/Amazon/computer”). We also have a variable to hold the base URL to API.AI (we discussed this in step 8 of creating our API.AI agent). Next, we store our API.AI Client Access Token in a variable named APIKEY, which will be passed to API.AI in the request header. Finally, we created an invocation word and prompt. Our continuous session will listen for the word “Hal,” at which time we will respond with “Yes?”, letting the user know we are listening and awaiting her next utterance:

   public sealed partial class MainPage : Page
   {
       // Speech Recognizer
       private SpeechRecognizer recognizer;
   
       private HttpClient client = new HttpClient();
       private readonly string baseURL = "https://api.api.ai/api/query?v=
         20150910&lang=en";
       private readonly string APIKEY = "092e4a9c21d34bed9e4f50e7ebf56d2d";
   
       private readonly string INVOCATION_NAME = "hal";
       private readonly string INVOCATION_PROMPT = "yes?";
   
       public MainPage()
       {
           this.InitializeComponent();
       }
       .......
   

9. The following code illustrates what occurs when our app is navigated to (similar to “loaded” in a traditional Windows app). Here, we create the Authorization Header to the Client Access Token so it is passed on each call to API.AI. Next, we call InitializeSpeechRecognizer to initialize the continuous recognition session:

   async protected override void OnNavigatedTo(NavigationEventArgs e)
   {
       base.OnNavigatedTo(e);
   
       client.DefaultRequestHeaders.Authorization = 
         new AuthenticationHeaderValue("Bearer", APIKEY);
       await InitializeSpeechRecognizer();
   }
           ....
   

10. Let’s break down the InitializeSpeechRecognizer method. After we create a new SpeechRecognizer object, we attach a delegate handler to the ContinuousRecgonitionSession’s ResultGenerated event. Next, we create a constraint for the session to only fire when certain words or phrases are received. In our case, we only want the event to fire by our invocation word “Hal.” Finally, we compile our constraints, and if successful, start the session:

    async protected override void OnNavigatedTo(NavigationEventArgs e)
    {
        base.OnNavigatedTo(e);
    
        client.DefaultRequestHeaders.Authorization = 
          new AuthenticationHeaderValue("Bearer", APIKEY);
        await InitializeSpeechRecognizer();
    }
    .....
    

11. Taking a quick look at the RecognizerResultGenerated event delegate we see that if the spoken word isn’t empty, we simply pass the value to a custom handler called ProcessContinuousVoiceResult:

    // Recognizer generated results
    async private void RecognizerResultGenerated(
      SpeechContinuousRecognitionSession session, 
      SpeechContinuousRecognitionResultGeneratedEventArgs args)
    {
        try
        {
            if (!string.IsNullOrEmpty(args.Result.Text))
            {
                await ProcessContinousVoiceResult(args.Result.Text);
            }
        }
        catch (Exception ex)
        {
            //log
        }
    }
    .....
    

12. Within ProcessContinuousVoiceResult we double-check that we received the invocation name, and if so, we respond with “Yes?”, letting our user know we are ready for her next phrase. Next, we stop the continuous speech recognizer session (temporarily) and call InitialIntentRecognizer to spin-up a second “burst” listening session:

    private async Task ProcessContinousVoiceResult(string target)
    {
        try
        {
            if (target.ToLower() == INVOCATION_NAME)
            {
                await this.Dispatcher.RunAsync(
                  Windows.UI.Core.CoreDispatcherPriority.Normal, 
                    () =>
                {
                    PlayResponse(INVOCATION_PROMPT);
                });
                await recognizer.ContinuousRecognitionSession.StopAsync();
                await InitializeIntentRecognizer();
    
            }
        }
        catch (Exception ex)
        {
            //log
        }
    }
    ....
    

13. The InitialIntentRecognizer is where most of the magic happens. The first part of the method creates a new SpeechRecognizer object, compiles any constraints (in our case there are none), and then sets up our timeout values. The timeout settings inform the session to wait up to 10 seconds for speech (and ending the session) and to wait 5 seconds after speech is heard. Next, we start the recognition session by calling RecognizeAysnc(). When the session has completed, we grab the utterance from the session result, build our query string, and pass the data to API.AI. Once the response is received from API.AI, we parse the data and pass the response speech to the PlayResponse method:

    private async Task InitializeIntentRecognizer()
    {
        string spokenWord = string.Empty;
    
        try
        {
            // Initialize recognizer
            using (var intentRecognizer = new SpeechRecognizer())
            {
                var compilationResult = 
                  await intentRecognizer.CompileConstraintsAsync();
                // If successful, display the recognition result.
                if (compilationResult.Status == 
                  SpeechRecognitionResultStatus.Success)
                {
                    // change default of 5 seconds
                    intentRecognizer.Timeouts.InitialSilenceTimeout =
                      TimeSpan.FromSeconds(10);
                    // change default of 0.5 seconds
                    intentRecognizer.Timeouts.EndSilenceTimeout =
                      TimeSpan.FromSeconds(5);
                    SpeechRecognitionResult result = 
                      await intentRecognizer.RecognizeAsync();
                    if (result.Status == 
                      SpeechRecognitionResultStatus.Success)
                    {
                        spokenWord = result.Text;
                    }
                }
            }
    
            if (!string.IsNullOrEmpty(spokenWord))
            {
                if (!string.IsNullOrEmpty(spokenWord))
                {
                    var result = await client.GetStringAsync(baseURL +
                      "&sessionId=" + Guid.NewGuid().ToString() + 
                            "&query=" + Uri.EscapeUriString(spokenWord));
                    var results = JObject.Parse(result);
                    var output = (string)results["result"]
                      ["fulfillment"]["speech"];
    
                    await this.Dispatcher.RunAsync(
                      Windows.UI.Core.CoreDispatcherPriority.Normal, () =>   
                   {
                        PlayResponse(output); 
                    });
                }
            }
        }
        catch (Exception ex)
        {
            //log
        }
        finally
        {
            //result the main recognition session to listen for trigger word
            await recognizer.ContinuousRecognitionSession.StartAsync();
        }
    }
    .....
    

14. Finally, let’s review the PlayResponse method. This method simply builds an SSML string, selects one of the available synthesized voices, and plays the SSML value via the MediaElement object:

    private async Task PlayResponse(string text)
    {
        try
        {
            MediaElement media = new MediaElement();
            SpeechSynthesisStream stream = null;
    
            string Ssml =
                @"<speak version="1.0">" +
                text +
                "</speak>";
    
            var voices = SpeechSynthesizer.AllVoices;
            using (var speech = new SpeechSynthesizer())
            {
                speech.Voice = voices.First(gender => gender.Gender == 
                  VoiceGender.Male && gender.Description.Contains("David"));
                stream = await speech.SynthesizeSsmlToStreamAsync(Ssml);
            }
    
            media.SetSource(stream, stream.ContentType);
            media.Play();
    
            media.Stop();
        }
        catch (Exception ex)
        {
            //log
        }
    }
    }
    .....
    

15. Now that we have reviewed the critical elements of our code, we are ready to take it for test drive. As mentioned earlier, you can choose to simply deploy and run our project to your IoT Core Pi, or copy and paste our code into your project.

    Note

    If you choose to copy and paste our code into your project, remember to replace the Client Access Token (APIKEY) to match your API.AI Token. If you choose to simply run our cloned project, you will need to update the Client Access Token as well the Remote Machine IP Address.

Deploying and Testing

Whether you use our project or build your own, VS makes deploying and testing the UWP app simple. Before we begin, let’s make sure our hardware is ready for testing. Ensure that your IoT Core device is powered up and the microphone and speaker are connected. Next, from the toolbar in VS, click the Remote Machine button. This will compile and deploy the app to your IoT Core device. Once the app is deployed (the VS toolbar options will update to display the Continue and Stop icons), you are ready to test the app.

While in debug mode, speak the word “Hal” into the microphone. You should hear a “Yes?” response from the device.  Once you hear “Yes?”, say the following into the mic: Say ‘Hello’. You should then hear the device reply with one of the response phrases created in API.AI. If everything worked as planned, pat yourself on the back—you just created a voice-activated app on IoT Core running on a Raspberry Pi!