This chapter serves as an introduction to the services in Microsoft Azure that support the most exciting part of the digital transformation wave that we are witnessing today.
Microsoft Azure, like all cloud providers, has its humble beginnings from IaaS, PaaS, and SaaS. In fact, these services continue to be important foundations and are still evolving, albeit with less fanfare compared to news about advancement in artificial intelligence (AI) and machine learning (ML). A field-programmable gate array (FPGA) is a good example of an infrastructure evolving to enable intelligence in cloud computing (see https://github.com/harris-soh-copeland-puca/azure-docs/blob/master/articles/machine-learning/how-to-deploy-fpga-web-service.md).
A large part of this second edition was rewritten with new content that was not present in the first edition, but that is not the only difference. Almost half the book has been dedicated to data, AI, machine learning, and advanced analytics. Two separate sections have been dedicated to these topics, respectively, and we have included more hands-on labs both in the book and added more to the book’s GitHub (https://github.com/harris-soh-copeland-puca). If you are already familiar with the data, AI and machine learning services in Azure and want to dive into specific topics, feel free to skip this chapter and go directly to the later chapters that dive deeper into the technology.
If you are looking for a high-level overview of the Azure services that are available in support of this emerging trend, or you are looking for examples of how Azure is used and deployed by data scientists, modern data engineers, and data-driven decision-makers, then this chapter was written with you in mind. We start with the services that consume data and make our way down to where modern data services provide the information foundation that makes it all happen.
Ethical AI
Before we get to the overview of the AI and machine learning capabilities in Microsoft Azure, it is important to take the time to discuss the topic of ethics as it pertains to the use of AI and the social issues related to AI.
Science Fiction and Reality: A Social Convergence
Hollywood science fiction has played its part in introducing intriguing effects of AI on society in movies like 2001’s AI: Artificial Intelligence and 2002’s Minority Report.
Both movies touch on extreme themes. In AI: Artificial Intelligence, audiences are left to ponder whether an artificial life form is worthy of the same protections as humans when an artificial child is built to possess emotions. In Minority Report, predictions made by AI are enforceable before the actual crime is committed.
As AI makes robots and digital assistants more human, the line between fiction and reality is indeed getting blurred. For example, contributing to Japan’s issue of an aging population is the generation of otakus,1 men who are in relationships with virtual girlfriends. A concept highly promoted in anime culture and gaming.
Machine learning, a subset of AI, uses algorithms to build mathematical models based on sample data for training and learning to make the most accurate predictions and decisions possible. How confident are we with machine-based decisions? Look around you at the Teslas, IoT devices, and smart homes. The US Navy and Boeing have a highly classified program (codenamed CLAWS) to develop AI-powered, Orca-class fully autonomous submarines that are outfitted with torpedoes, meaning these submarines can decide to kill indiscriminately. The obvious ethical question is, should a machine be able to kill without human intervention? The increase in concern regarding the need, or potential lack thereof, human intervention in AI decision making has led to the alternative, and growing popularity of not calling the acronym AI artificial intelligence, but rather augmented intelligence.
Another sign of the times is the explosion of fake news. Advancements in AI and machine learning are making it easier to create extremely convincing fake videos, known as deepfake. Experts fear that deepfake videos will create an even larger trust issue in journalism. These videos are also being employed in politics.
What Is Ethical AI?
Ethical AI is a study and governance surrounding the use of AI technologies in a way that does not violate human rights and endanger lives. We are not exactly at the point of having to address the rights of robots, so that part is still the domain of science fiction.
The impact of AI on life is so important. As we rapidly meet and surpass human parity in many areas, religious organizations have played a leading role in addressing this issue proactively. In February 2020, the Vatican in Rome released a guideline for ethical AI called “Rome Call for AI Ethics,”2 to which Microsoft is one of the first to sign. Listen to Microsoft President Brad Smith’s speech at https://romecall.org.
There are many discussions and representations of what ethical AI means, but the three principles in “Rome Call for AI Ethics” summarizes the core concepts: ethics, education, and rights.
AI-based technology must never be used to exploit people in any way, especially those who are most vulnerable. Instead, it must be used to help people develop their abilities (empowerment/enablement) and to support the planet.
—The Vatican
Microsoft AI Principles
An important hallmark of our approach to responsible AI is having [the] ecosystem to operationalize responsible AI across the company, rather than a single organization or individual leading this work.
—Microsoft
Fairness: AI systems must treat all people fairly and with no bias to age, culture, gender, or national origin.
Inclusiveness: AI systems must engage and empower everyone, with no bias to age, culture, gender, or national origin.
Reliability and safety: AI systems should perform reliably and safely.
Transparency: AI systems should be understandable.
Privacy and security: AI systems should be secure, protect, and respect privacy.
Accountability: AI systems should have algorithmic accountability.
We dedicated a generous amount of time discussing ethical AI because it is a very important one as the technology evolves. For the rest of this chapter, we introduce the different AI technologies from Microsoft and look at some use cases. However, where applicable, we bring up specific ethical and legal considerations as it relates to the services.
Microsoft Cognitive Services
Microsoft Cognitive Services is a class of services that helps us build services that can interact with users more humanly. It is part of the AI family of capabilities. The goal of Azure Cognitive Services is to help developers create applications that can see, hear, speak, understand, and even begin to reason. The catalog of services within Azure Cognitive Services can be categorized into five main pillars: vision, speech, language, web search, and decision. Find out more at https://github.com/harris-soh-copeland-puca/azure-docs/blob/master/articles/cognitive-services/Welcome.md.
To make applications more human-like, we need to enable capabilities that go beyond how users traditionally interact with applications. That interaction was limited primarily to the lack of cognition. After all, it was only recently that computers could respond to spoken and understand spoken commands.
Measuring the advancement of computer cognitive abilities is based on test results against human performance. Where those results meet or surpass the capabilities of a human, then we would say we have reached human parity. So, how does a computer reach human parity for a capability? Humans make mistakes, such as incorrectly identifying something visually or did not hear certain words spoken in a sentence, so reaching human parity must take this into account. It does not mean the computer needs to be 100% accurate or have a 0% error rate. We provide examples of this comparison as we discuss each of the cognitive abilities.
Object Recognition
The advancement in AI at Microsoft culminated in an explosion of breakthroughs in the past two years. In the area of Computer Vision, Microsoft achieved human parity in 2016 by accurately identifying all the objects in a visual input with an accuracy of 96% compared to humans at 94%. Using a deep neural network called a residual neural network (ResNet) to train computers to recognize images, Microsoft went five times deeper than most other attempts to get the result (152 layers compared to 20 to 30 layers). Today, with the number of images available and the computing resources, computer vision is at 99% accuracy.
Through custom computer vision, we can train Microsoft’s Vision cognitive services to identify images that fit the business need. The Custom Vision service uses a machine learning algorithm to apply your labels to images and hence provide customized image classifiers. To find out more about Custom Vision, visit https://github.com/harris-soh-copeland-puca/azure-docs/blob/master/articles/cognitive-services/Custom-Vision-Service/home.md.
Use Case Scenarios
Many use cases for object detection in Custom Vision apply in a public safety application. For example, images from the video feed of a security camera in a park setting can train a model to detect firearms. When a possible firearm is detected, which is not an object that should be in a park, law enforcement can be notified immediately.
In another example, a model can be trained to recognize wheelchairs. We can then provide medical facilities with the ability to quickly locate wheelchairs by accessing AI-driven cameras based on custom vision and object detection.
Face AI
Detect a face (face detection)
Recognize a face (face verification)
Detect emotion (perceived emotion recognition)
Face Detection vs. Face Recognition
Face detection is a specific form of object recognition in that the API is tuned to detect faces from a picture or video. It is important to differentiate face detection from face verification primarily because the latter falls under more stringent personally identifiable information (PII) protection.
Detecting a face does not equate to recognizing a face, but it is important to validate laws covering facial AI use are drafted to address this distinction. For example, in May 2019, San Francisco became the first US city to ban the use of facial recognition technology by police and city government agencies (see www.npr.org/2019/05/14/723193785/san-francisco-considers-ban-on-governments-use-of-facial-recognition-technology). In the same year, the American Civil Liberties Union (ACLU) sued multiple government organizations requesting the release and use of facial recognition (www.cnn.com/2019/11/01/us/aclu-sues-federal-government-over-surveillance-from-facial-recognition-technology/index.html).
Facial verification tests
Face detection tests
Tests for perceived emotion detection
Use Case Scenarios
A use case introduced at a hackathon that we facilitated involved patient intake at hospitals. In this scenario, it is not uncommon to have patients who are unconscious and without ID. If these patients have been treated at the facility before, facial recognition will help emergency personnel identify the patient and their medical history to render treatment expeditiously.
Another use case can be one in a classroom setting. Perceived emotions from students can help teachers determine their emotional states, especially if the students are very young children who may not be comfortable voicing their feelings in the classroom.
Lastly, facial recognition and detection are widely used in public safety scenarios, as seen in the sample tests we conducted earlier.
Speech Services3
In 2017, Microsoft reached a 5.1%-word error rate. This means that for a transcription session, Microsoft’s speech recognition can correctly transcribe 94.9% of the words spoken in an audio input. Compare this to the error rate of professional transcriptionists, which is 5.9%, as measured by the switchboard portion of the NIST 2000 test set where pairs of strangers discuss an assigned topic. At a 5.1%-word error rate, Microsoft’s speech recognition reached4 and surpassed human parity. Microsoft Speech service is the unification of speech to text, text to speech, speech translation, and speaker recognition.
Speech to Text
Speech to Text is a straightforward service. The two main scenarios for consuming this service is real-time transcription and post-transcription using a recorded audio file. At the time of writing, audio files must be .wav files. In our GitHub repo, we publish a project that accepts an incoming .wav file and translates it to text. This use case, dubbed CaseTalk, is based on a scenario where social workers visit rural communities that may not have good cellular and Internet connectivity.
These social workers record their interviews with their wards on mobile devices or laptops and later transcribe the recordings. Since the social workers’ time includes the commute to remote locations, the transcribed notes are not updated in a timely fashion. Such delays lead to long turnaround times to get aid to the community, and in some cases, entering potentially life-threatening situations. A simple speech-to-text interface was created to allow social and caseworkers to upload voice recordings and subsequently convert those to text via the speech-to-text API. This reduces the amount of work for caseworkers and reduces the time to get the appropriate assistance for community members.
Text to Speech
Advancement in text-to-speech technology allows computers to sound more human and in staccato and disjointed sounding sentences.
Microsoft has been investing in AI research for at least 25 years, with much of that research surrounding text to speech. For this cognitive skill, we thought it would be fun to demonstrate how computers sounded before. Some of us may remember the Apple II and Commodore 64 computers, and back then, there was a small text-to-speech program written in C called Software Automatic Mouth (SAM). So, we went looking for SAM and were pleasantly surprised to find a project by Sebastian Macke. He ported SAM so that it can be used in a web browser (see https://simulationcorner.net/index.php?page=sam).
Today, natural-sounding text-to-speech capabilities make following instructions provided by a computer a lot easier, increases the ability for call centers to push more functionality to automated attendants, and even appear on TV. See the world’s first AI news anchor at www.cnbc.com/2018/11/09/the-worlds-first-ai-news-anchor-has-gone-live-in-china.html.
Speaker Recognition
Speaker recognition is the most recent addition to speech services, and at the time of writing, it is still in preview. It allows us to train using the voice of different speakers and subsequently identify the different speakers in a setting.
One use case scenario that this is applied to is speaker diarization in a courtroom, interview, or conference setting, where there are many known speakers and a need to identify who said what during the event. Speech diarization with speaker recognition can completely automate the role of a court transcriptionist. Speaker recognition allows post-processing of audio streams, which provides the ability to quickly search for things said by individuals.
Machine Reading Comprehension
In 2018, using the Stanford Question Answering Dataset (SQuAD), which is a reading comprehension dataset consisting of over 100,000 questions and answers based on Wikipedia articles, Microsoft reached an exact match (EM) score of 82.650%. The SQuAD dataset consists of 50,000 unanswerable questions, which were written to look like answerable ones. Scoring is based on a system’s ability to answer questions when possible and to determine which are unanswerable questions and abstain from providing one. These scores are higher than the best human score, which is 82.304%, and therefore human parity was reached.
Machine Translation
A machine translation (MT) system is a data-driven system that combines rule-based and statistical methods into a hybrid system that is then capable of learning lexical and phrasal translations from the data derived from different languages.
Human parity translation by language
Use Case Scenarios
Imagine a scenario where there are more than two languages spoken in a single setting. Perhaps an international courtroom like the International Court of Justice, also known as the World Court. Cases involve multiple countries with 15 judges from different parts of the world. It is fathomable that more than one interpreter might simultaneously be needed because there may be more than one or two languages spoken during hearings.
Another use case scenario can be made for multinational organizations. With ecommerce, this is no longer the domain of large companies. Even small companies have global customers s and require translation—spoken and written—in many languages. A single AI service that provides multilanguage translations is more efficient, timely, and cost-effective than traditional means.
Text Analytics: Sentiment
Aside from transcription, translation, and understanding text, there is also sentiment. Sentiment describes the emotional aspect of a body of text, which would reflect the opinion and feelings of the author. Extracting sentiment out of text through AI provides more accurate responses, as Referenced in Figure 5-5. For example, if we have a stack of written reports to review, we may want the ability to prioritize by sentiment would ensure that the most negative reports are addressed first, assuming there are significant issues that warrant immediate action. The API provides advanced natural language processing over raw text and includes four main functions: sentiment analysis, key phrase extraction, language detection, and named entity recognition.
Extracting sentiment from a body of text at https://azure.microsoft.com/en-us/services/cognitive-services/text-analytics
Bots
All the cognitive services mentioned so far are used in digital assistants like Bots to communicate with users via a variety of channels like the web, Teams, SharePoint, and even voice (forthcoming). More than 300,000 developers are creating digital agents with Azure Bot Service.
Use Case Scenarios
In 2020, the world was in the midst of a novel coronavirus pandemic that shut down or overwhelmed many call centers. Bots were deployed at a record pace to help alleviate some of the load and efficiently get accurate information to citizens.
On a lighter note, there is a great case study by an esteemed colleague of ours, Eric Egland, an AI specialist, and his work with the State of California and the Eureka Bot. Eureka is the product of the Secretary of State’s digital initiative to modernize and digitize the agency to make it easier to do business in California, and shortly after its launch in 2019, it has served more than 77,000 inquires. Most of these inquiries were offloaded from call centers. You can read the full story at https://cloudblogs.microsoft.com/industry-blog/government/2019/08/15/eureka-chatbot-helps-california-stay-open-for-business/.
There are many other AI and cognitive services such as video, search, anomaly detection, forms recognizer, and so forth. New capabilities are added, and existing ones improved upon on a very regular basis. Visit the Azure Cognitive Services page at https://azure.microsoft.com/en-us/services/cognitive-services/#features. This is one of our favorite pages because you can interactively try out all the cognitive services on the website.
Azure Machine Learning
Machine learning is a subset of AI. We cover it in detail in Chapter 16. The advancement in all the cognitive services mentioned in the previous section is based on Machine learning and the vast amount of available training data.
Machine learning trains models to not only recognize or detect, but also to predict outcomes and thus help drive decision making. Machine learning is fundamentally based on data science and statistical methods. The amount of data and statistical methods often requires a lot of memory and compute, which is an ideal fit for Azure.
Comparing Azure Machine Learning and Azure Machine Learning Studio (Classic)
|
Azure Machine Learning
Azure Machine Learning is a cloud-based environment for data scientists to train, deploy, automate, manage, and track ML models. It is often used in conjunction with cloud storage such as Azure Blob Storage and Azure Data Lake Store because of the potentially large amount of data being used in the ML process.
Azure Machine Learning supports all the tools and languages used by data scientists, such as R and Python. It also has tools such as Jupyter notebooks and supports open source add-ons such as PyTorch, TensorFlow, scikit-learn, and YOLO. It comes with a web interface called the Azure Machine Learning designer.
Azure Machine Learning supports classical ML, deep learning, as well as supervised and unsupervised learning. Azure Machine Learning also integrates with other Azure services, such as Azure DevOps, to help secure the data scientists’ work through source control.
Machine Learning Studio (Classic)
Azure Machine Learning Studio (classic) is the initial implementation and ML offering in Azure. It is still available and should not be confused with Azure Machine Learning. See the differences in Table 5-1. Download the Azure Machine Learning Studio (classic) architecture to see all its capabilities at https://download.microsoft.com/download/C/4/6/C4606116-522F-428A-BE04-B6D3213E9E52/ml_studio_overview_v1.1.pdf.
Azure Databricks
Azure Databricks is the native implementation of Databricks in Azure and is provided as a PaaS offering. Azure Databricks provides all the tools and resources that data scientists and researchers need and is jointly managed by Databricks and Microsoft engineers.
Azure Databricks is provisioned as workspaces that contain the customizable compute clusters, the notebooks, datasets, and storage. Users can use Databricks File System (DFS) or mount external storage like Azure Blob Storage or Azure Data Lake Store to access data for projects. Azure Databricks notebooks allow users with different skillsets to collaborate because the notebooks support different languages like Python, R, Scala, and SQL. All notebooks can be source controlled using GitHub or Azure DevOps.
There are two types of compute clusters in Azure Databricks: default and high concurrency clusters. To optimize resources and reduce costs, the default compute clusters in Azure Databricks can automatically shut down due to inactivity. This threshold is set at 120 minutes, but this can be configured as needed. High concurrency clusters are configured to not terminate automatically. Visit https://docs.microsoft.com/en-us/azure/databricks/clusters/configure to find out more about cluster configuration.
Compute clusters can be customized with the desired libraries needed, such as PyTorch, scikit-learn, and TensorFlow, and so forth. Compute clusters can be pinned so that the customization is preserved even after the cluster is shut down due to inactivity.
Use Cases for Azure Databricks
Azure Databricks prepares raw data for downstream systems and analysis. Azure Databricks is a collaborative tool with a target audience of data scientists, data engineers, researchers, and business intelligence analysts.
Data ingestion and manipulation pipeline involving Azure Databricks
Azure Data Science Virtual Machines
In the most traditional approach, data scientists can always use regular virtual machines as their environment. This approach relies on the data scientist to completely customize a personally provisioned virtual machine with all the tools and libraries needed. Usually, it relies on the same virtual machine for compute and storage. It is not the most efficient and scalable model, but it is a common setup that is still widely used.
Microsoft Azure provides the data scientists that work in this setup with specialized virtual machine images in the gallery. It comes with preinstalled tools that are generally used in data science. This type of image is called a Data Science Virtual Machine (DSVM). Data scientists should choose this option over on-premises virtual machines or standard virtual machine images in Azure because of the specialized VM image and the ability to scale compute and storage resources for the VM as needed. Azure DSVMs are available with Linux or Windows operating systems. DSVMs can also be attached as compute to Azure Notebooks or in Azure Machine Learning Service to expand the compute power.
Summary
In this chapter, we covered the important topic of ethics as it pertains to AI. We also provided a high-level introduction to the AI capabilities in Azure, such as cognitive services and machine learning. We looked at tooling options available to data scientists and researchers as such Azure Databricks and the Data Science Virtual Machines. All these topics are covered in greater detail in Chapters 15 and 16. Additional documentation for Cognitive Services is on our GitHub repo at https://github.com/harris-soh-copeland-puca/azure-docs/tree/master/articles/cognitive-services. Documentation for Azure Machine Learning is located at https://github.com/harris-soh-copeland-puca/azure-docs/tree/master/articles/machine-learning. Documentation for Azure Databricks is located at https://github.com/harris-soh-copeland-puca/azure-docs/tree/master/articles/azure-databricks.