The first step we need to do is to add an enum variable containing the intents. Create a new file called LuisActions.cs, in the Model folder, and add the following content to it:

    public enum LuisActions {
        None, GetRoomTemperature, SetRoomTemperature, GetNews
    }

If you have any other intents defined, add them as well.

This enum will be used later, to see which action to execute when triggered. For instance, if we ask to get the latest sports news, GetNews will be triggered, which will go on to retrieve news.

To make things a bit easier for ourselves, we are going to use the existing LUIS example for the rest of the chapter. An alternative would be to add this to the HomeView, where we could continuously listen to spoken commands from the users.

In order to trigger an action, we need to open the LuisViewModel.cs file. Find the OnLuisUtteranceResultUpdated function. Let us update it to the following:

    private void OnLuisUtteranceResultUpdated(object sender, LuisUtteranceResultEventArgs e)
    {
        Application.Current.Dispatcher.Invoke(async () => {
            StringBuilder sb = new StringBuilder(ResultText);
                
            _requiresResponse = e.RequiresReply;

            sb.AppendFormat("Status: {0}
", e.Status);
            sb.AppendFormat("Summary: {0}

", e.Message);

At this time, we have not added anything new. We have removed the output of entities, as we do not need this anymore.

If we find that any actions have been triggered, we want to do something. We call a new function, TriggerActionExecution, passing on the name of the intent as a parameter:

    if (!string.IsNullOrEmpty(e.IntentName))
        await TriggerActionExectution(e.IntentName, e.EntityName);

We will get back to this function shortly.

Complete OnLuisUtteranceResultUpdated by adding the following code:

            ResultText = sb.ToString();
        }); 
    }

Again, you should see that there are no new features. We have, however, removed the last else clause. We do not want to have the application speak the summary to us anymore.

Create the new TriggerActionExecution function. Let it accept a string as the parameter, and have it return a Task. Mark the function as async:

    private async Task TriggerActionExectution(string intentName) {
        LuisActions action;
        if (!Enum.TryParse(intentName, true, out action))
            return;

Here, we parse the actionName (intent name). If we have not defined the action, we will not do anything else.

With an action defined, we go into a switch statement to decide what to do. As we are only interested in the GetNews case, we break out from the other options:

        switch(action) {
            case LuisActions.GetRoomTemperature:
            case LuisActions.SetRoomTemperature:
            case LuisActions.None:
            default:
                break;
            case LuisActions.GetNews:
          break;
        }
    }

Make sure that the code compiles before continuing.

Next, we will need to modify the Luis.cs file. As we have defined an entity for the news topic, we want to ensure that we get this value from the LUIS response.

Add a new property to LuisUtteranceResultEventArgs:

    public string EntityName { get; set; }

This will allow us to add the news topic value, if received.

We need to add this value. Locate ProcessResult in the Luis class. Modify the if check to look like the following:

        if (!string.IsNullOrEmpty(result.TopScoringIntent.Name)) {
            var intentName = result.TopScoringIntent.Name;
            args.IntentName = intentName;
        }

        else {
            args.IntentName = string.Empty;
        }

        if(result.Entities.Count > 0) {
        var entity = result.Entities.First().Value;

        if(entity.Count > 0)  {
            var entityName = entity.First().Value;
            args.EntityName = entityName;
        }
    }

We make sure that the intent name, of the top-scoring intent, is set, and pass it on as an argument to the event. We also check if there is any entities set, and if so, pass on the first one. In a real-life application, you would probably check other entities as well.

Back into the LuisViewModel.cs file, we can now account for this new property. Let the TriggerActionExecution method accept a second string parameter. When calling the function, we can add the following parameter:

    await TriggerActionExectution(e.IntentName, e.EntityName);

To be able to search for news, we need to add a new member of the BingSearch type. This is the class we created in the previous chapter:

    private BingSearch _bingSearch;

Create the object in the constructor.

Now we can create a new function, called GetLatestNews. This should accept a string as the parameter, and return Task. Mark the function as async:

private async Task GetLatestNews(string queryString)
{
    BingNewsResponse news = await _bingSearch.SearchNews (queryString, SafeSearch.Moderate);

    if (news.value == null || news.value.Length == 0)
        return;

When this function is called, we SearchNews on the newly created _bingSearch object. We pass on the queryString, which will be the action parameter, as the parameter. We also set the safe search to Moderate.

A successful API call will result in a BingNewsResponse object, which will contain an array of news articles. We are not going into more details on this class, as we covered it in Chapter 9, Adding Specialized Searches.

If no news is found, we simply return from the function. If we do find news, we do the following:

    await ParseNews(news.value[0]);

We call a function, ParseNews, which we will get back to in a bit. We pass on the first news article, which will be parsed. Ideally, we would go through all the results, but for our case, this is enough to illustrate the point.

The ParseNews method should be marked as async. It should have the return type Task, and accept a parameter of type Value:

private async Task ParseNews(Value newsArticle)  {
    string articleDescription = $"{newsArticle.name}, published {newsArticle.datePublished}. Description:
    {newsArticle.description}. ";

    await _ttsClient.SpeakAsync(articleDescription, CancellationToken.None);
}

We create a string containing the headline, the publishing date, and the news description. Using this, we call SpeakAsync on the _ttsClient to have the application read the information back to us.

With this function in place, we can execute the action. In TriggerActionExecuted, call GetLatestNews from the GetNews case. Make sure to await the call.

With the application compiling, we can go for a test run:

Naturally, the effects are not as good in an image as in real life. With a microphone and speakers or headset connected, we can ask for the latest news, using audio, and get the news read back to us with audio.

News articles often come with corresponding images as well. As an addition to what we already have, we can add image analysis.

The first step we need to do is to add a new NuGet package. Search for the Microsoft.ProjectOxford.Vision package, and install this using NuGet Package Manager.

In the LuisViewModel.cs file, add the following new member:

private IVisionServiceClient _visionClient;

This can be created in the constructor:

_visionClient = new VisionServiceClient("FACE_API_KEY", "ROOT_URI");

This member will be our access point to the Computer Vision API.

We want to get a string describing the image in the ParseNews function. We can achieve this by adding a new function, called GetImageDescription. This should accept a string parameter, which will be the image URL. The function should have return type Task<string> and be marked as async:

private async Task<string> GetImageDescription(string contentUrl)
{
    try {
        AnalysisResult imageAnalysisResult = await _visionClient.AnalyzeImageAsync(contentUrl, new List<VisualFeature>() { VisualFeature.Description });

In this function, we call AnalyzeImageAsync on the _visionClient. We want the image description, so we specify this in a list of VisualFeature. If the call succeeds, we expect an object of type AnalysisResult. This should contain image descriptions, ordered by probability of correctness.

If we do not get any descriptions, we return none. If we do have any descriptions, we return the text of the first one:

    if (imageAnalysisResult == null || imageAnalysisResult.Description?.Captions?.Length == 0) 
        return "none";
    return imageAnalysisResult.Description.Captions.First().Text;
}

If any exceptions occur, we print the exception message to the debug console. We also return none to the caller:

        catch(Exception ex) {
            Debug.WriteLine(ex.Message);
            return "none";
        }
    }

In ParseNews, we can get the image description by adding the following at the top of the function:

string imageDescription = await GetImageDescription (newsArticle.image.thumbnail.contentUrl);

With an image description, we can modify the articleDescription string to the following:

    string articleDescription = $"{newsArticle.name}, published
           {newsArticle.datePublished}. Description:
           {newsArticle.description}. Corresponding image is      
           {imageDescription}";

Running the application and asking for news will now also describe any images. That concludes our smart-house application.