Introduction to AWS IoT

AWS IoT is Amazon’s cloud-based IoT platform. Its main purpose is to gather data from a myriad of devices into the cloud for further processing in a scalable fashion. AWS IoT supports the following key use cases, as depicted in Figure 2-1:

Communicate

Enables devices to securely communicate with cloud applications and each other using HTTP or MQTT (directly, or via WebSocket) protocols. AWS IoT may act as a protocol bridge, so devices can talk to each other even if they don’t quack the same protocol. The communication is usually bidirectional; devices deliver their telemetry data into the cloud, and receive control commands from the cloud.

Execute Data Analytics

This is the core strength of AWS IoT, since data may be filtered, transformed, and acted upon using AWS’s scalable and rich set of services. AWS IoT may be interrelated with services, like Amazon Simple Notification Service, Amazon Simple Queue Service, Amazon Simple Storage Service, Amazon DynamoDB, Amazon Kinesis, Amazon Machine Learning, and AWS Lambda. More fine-tuned views can be implemented by leveraging Amazon Athena and Amazon QuickSight. Such higher-level intelligences are invaluable for better decision making and supervision of remote processes.

Manage Device

AWS IoT buttresses device shadowing, which emulates a virtual live device for applications. Under the hood, AWS IoT stores the latest state of a device, and exposes it for read/write operations. Applications are oblivious as to whether the target device is online or offline. In the latter case, the AWS IoT will sync the virtual state with the device once it appears online.

Figure 2-1. Major use cases and actors of AWS IoT. These use cases are applicable across many IoT platforms, so they are rather universal.

All interactions between actors and uses cases in Figure 2-1 happen in a secure manner. AWS IoT relies on certificates to authenticate parties, and uses AWS’s security features to enforce authorization. All external communication channels are encrypted for ensuring confidentiality, integrity, and privacy of data.

Overview of the Architecture

Figure 2-2 shows the major constituent elements of an AWS IoT solution:

Device (Thing)

This is the component (for example, sensor, actuator, smart appliance, microcontroller) that interacts with a physical environment, and is interconnected to the AWS cloud. The device hosts a simple embedded application, which uses the AWS IoT Device SDK to send/receive messages to/from AWS IoT.

AWS IoT Edge (Greengrass stack)

AWS Greengrass is a reduced AWS ecosystem close to the matching set of devices. The AWS Greengrass consists of the Greengrass core software (engine) and the associated SDK. This core may be deployed on a node-neighboring target device. The pertinent devices must form a Greengrass group with this dedicated node. The group appears as a unified device inside AWS IoT. Devices may interact inside this group even while it is disconnected from the cloud. The Greengrass core supports the AWS Lambda facility, so it is possible to develop applications to react on locally important events, or filter traffic between devices and the cloud.

Device Gateway

This is the entry point into the AWS IoT. It also acts as a protocol bridge between devices.

Message Broker

Implements the necessary infrastructure for the publish/subscribe communication pattern between devices and AWS applications. Messages may be published via the MQTT or REST protocols, and consumed via the MQTT protocol.

Rules Engine

Filters, transforms, and reacts to messages emanated from devices. Its role is to decide what happens next based upon the message payload (it supports an SQL-like language to peek into the content). One possibility is to forward the message further down the AWS chain, and/or to emit a new message via the message broker to relevant subscribers.

Security Layer

Contains common security-related services to realize authentication, authorization, and secure communication between parties (both externally and internally).

Device Registry

Stores pertinent resources interrelated with a device (like certificates, MQTT client identifiers, custom attributes). Registering your devices enhances security, and lessens the overall device management burden.

Device Shadow & Service

Stores device state inside the cloud, and manages that state. Thanks to this service, a virtual device may appear for applications as a continuously online device.

AWS and Applications

These are all the components that exist in the AWS irrespective of AWS IoT. The only exception is the class of AWS IoT–aware applications that use the AWS IoT API or SDK.

Figure 2-2. The architecture of the AWS IoT solution

The elements in Figure 2-2 are grouped, depending on where data processing occurs, into the following categories:

• Device

• AWS IoT Edge

• AWS IoT

• AWS and applications

The ordering of groups is fundamental to understand scalability. Each bears a separate feedback loop, from fastest (device) toward slowest (AWS and applications). Let’s take a look at a concrete example to better understand this. Suppose you have a smart house equipped with smoke detectors and sprinklers managed by a micro-controller (it could perform as a group coordinator). Here are the main responsibilities of each group for this circumstance:

Smoke Detector (Device)

Send regular health status information toward the group coordinator. Detect smoke. In case of smoke, turn on the siren on the detector, and publish an alarm message.

Sprinkler (Device)

Send regular health status information toward the group coordinator. Listen to published smoke alarm messages. In case of a smoke alarm, open a valve to discharge water.

Greengrass Group Coordinator (AWS IoT Edge)

Monitor the health status of devices. In case of device malfunction, send an alert message toward AWS IoT. Function as a message broker. In case of a smoke alarm, forward the message toward AWS IoT.

AWS IoT

In case of a smoke alarm, initiate an automatic call to the fire department, and send a text message to the house owner. In case of a device failure, send a text message to the house owner. Persist unusual events into Amazon DynamoDB.

AWS and applications

Provide various reports about unusual events.

Figure 2-3 shows a sequence of actions in case of a fire. Local actions execute much faster than background tasks inside the cloud. Notice that opening a valve happens in parallel to the stuff inside the AWS cloud. The cloud-induced latency can be critical in some use cases. For example, it would be too late to inform yourself about a fire by looking at the database report.

Figure 2-3. The sequence of actions in case of a fire

Setting Up the Ecosystem

Once you are logged into your AWS account, you should choose the AWS IoT service from the dashboard.¹ You should see the AWS IoT console (Figure 2-4). We highly recommend you select the tutorial at this moment. It contains nice interactive animations that explain the major architectural elements of the system. Figure 2-5 shows the steps to initialize the AWS IoT with a single device (in our case, we will configure and test a contrived smoke detector).² During the progression AWS IoT will automatically set up the device gateway, and associate its endpoint with your account (this only occurs once).

Figure 2-4. The AWS IoT Console

Figure 2-5. The major steps to initialize AWS IoT with a device

arn:aws:iot:your-region:your-aws-account:
    resource/resource-identifier

Configure a Device and Setup Rules

Your device must possess the security files that you’ve downloaded previously. These are needed for authentication with AWS IoT. This step is highly device dependent (for the AWS IoT Button it is well elaborated in the AWS IoT documentation).

At this point you should test the whole setup. The easiest way is to monitor the relevant topic for messages published by your device. AWS IoT comes bundled with a simple MQTT client.³ Choose Test in the left pane. In the upper part of the page, enter “smoke” in the Subscription topic field, and press the button to subscribe to this topic. Select the topic on the left side, and fill in the publishing section as depicted on Figure 2-6. When you press the button to publish a message, it should appear in the lower part of the screen. Of course, simulating such a message publication is only necessary when you don’t have a physical device.

The final stage is to set up rules to perform pertinent actions when events occur. In our case, we would like to store the event inside Amazon DynamoDB (we are following the flow from Figure 2-3). Every rule is associated with a triggering condition, which is based on the message content. As depicted in Figure 2-6, the field status has a value of alarm in case of a fire. Another possible content is failure, which is sent by the coordinator if it detects missing messages, or receives a message indicating a problem with the detector (for example, when the detector emits a warning about low battery power). As an exercise, you may want to add a second action that would also send a text message to your mobile phone (see the AWS IoT documentation about setting up Amazon SNS with the rules engine). Keep in mind that a rule may contain many actions.

Figure 2-6. Testing message publication via the AWS MQTT client

A rule may also be configured with an IAM role. An IAM role contains IAM policies that specify the permissions for the rule’s action regarding AWS services. This is very similar to the policies you have seen earlier in this section. It is wise to reduce the rule’s capabilities, and only allow access to needed AWS services.

Choose Act on the left pane, and press the button to create a new rule. Table 2-3 shows the required values of the fields from the page.

Table 2-3. Values for fields to define a new rule

Field name	Field value	Comment
Name	PersistSmokeDetectorEvent	The rule’s unique name.
Description	Stores a critical smoke detector event inside the database for further reporting purposes	An explanation of what this rule performs.
Using SQL version	latest stable version	Select the latest stable version number. This specifies the version of the SQL rules engine. Be careful with the “beta” version.
Attribute	*	Determines what will be stored in the database. The * means all the fields from the JSON message.
Topic filter	smoke	The name of the topic to monitor.
Condition	status = ‘alarm’ or status = ‘failure’	The triggering condition. Here, it is a message with these values for the status field. We don’t want to store OK messages.

Now, choose the button to add an action. On the next page select the item for inserting a new record into Amazon DynamoDB, and press the button to configure the action. On the next page, you must first create a database resource (unless you already have a proper table). This will open the AWS dashboard of DynamoDB. Choose to create a new table. Give it the name SmokeDetectorEvents, and set the client‚Id field as the primary key. Leave all the other options intact, and press the button to create the table.

Note

The client identifier is set by the device, and is a standard part of the MQTT protocol message. The database primary key must spread the content uniformly, which is a prerequisite to attain scalability.

You should then return to the page where you can configure an action. Select the newly created table from the drop-down box (press the refresh button next to it if it doesn’t show up). Specify ${clientId} as the hash key value. Press the button to create an IAM role, and give it a name of DDBSmokeDetectorRole. Create the role, and select it from the drop-down box. You now need to choose to update the role for this rule. Finally, press the button to add the action, and afterward to create the rule.

Now, try to publish the following message with the MQTT client on the smoke topic:

{
  "clientId" : "1234",
  "status": "alarm",
  "chemical" : "smoke",
  "level" : "high"
}

Figure 2-7 shows the content of the database after performing the query to select the above message. At this point, you’ve successfully set up the AWS IoT ecosystem. You could also automate these steps using AWS CLI or AWS IoT API.

Figure 2-7. The content of the SmokeDetectorEvents table

Event Management and System Dashboards

AWS IoT has a built-in dashboard for monitoring messages. Choose Monitor from the left pane. The monitor’s page is divided into the following sections:

• Successful connections

• Messages

• Message published

• Rules executed

• Shadow updates

The granularity of the reports may be selected from the drop-down list in the upper-right corner of the screen. Figure 2-8 shows the section about messages.

Figure 2-8. The statistics about messages

In production, you should monitor AWS IoT using the AWS CloudWatch facility. It gathers and transforms raw data from AWS IoT into comprehensible metrics. Moreover, you may create CloudWatch alarms that are triggered when something goes amiss in the system. In the previous section, we mentioned that missing messages from a smoke detector is one failure case. Instead of handling these edge cases programmatically you can set up a CloudWatch alarm to take care of this (for example, by sending a notification via Amazon SNS). See the AWS IoT documentation for more details.

Application Integration Layer

Applications may use the whole spectrum of AWS services including AWS IoT. AWS offers the following types of APIs:

• AWS command-line interface (CLI)

• AWS REST API

• AWS SDK (language-specific library for emulating the web API)

The AWS IoT API exposes management interfaces toward devices as well as other elements of the AWS IoT service. Data analytics is done by the rest of the AWS world. For a good overview of AWS, including security, you may watch the webinar “Amazon Web Services AWS: AWS Fundamentals” by Richard A. Jones (Addison-Wesley Professional).

One potent way to build distributed applications with AWS is to leverage AWS Step Functions. This allows you to orchestrate components of distributed applications and microservices using visual workflows. It perfectly integrates in scalable fashion with the AWS Lambda service, which is the primary vehicle for handling asynchronous events in AWS IoT. For a complete example, refer to the article “Implementing Dynamic ETL Pipelines Using AWS Step Functions” by Tara Van Unen and Wangechi Doble (Amazon).

Security

You’ve already seen the important elements regarding AWS IoT security in “Setting Up the Ecosystem”. Figure 2-9 is the simplified UML class diagram showing how security-related artifacts are associated with devices and rules. For a more thorough treatment of securing AWS, check out Amazon Web Services in Action by Michael Wittig and Andreas Wittig (Manning). This book also introduces you to the full spectrum of AWS services.

Figure 2-9. The major elements of the AWS IoT security model

Building an End-to-End Example

Suppose you need to build a modern IoT-based system to support manufacturing. One of the major use cases is to predict machine failures, to enable scheduling maintenance work in advance to avoid unexpected downtimes. Machines are equipped with all sorts of sensors, and these generate lots of messages. Classical solutions cannot cope with such an abundance of data. The new system should provide the following features:

• Real-time monitoring and acting on events

• Historical static reports boosted with BI analysis

• Raw and aggregated historical data enabling ad hoc analysis

The power of IoT stems from the combination of event sources, and an ability to process them in a performant manner. Figure 2-10 illustrates a potential architecture that is based on edge computing and the lambda architecture pattern (don’t confuse this with the AWS Lambda service).

The Greengrass component handles fast feedback loops to react to high-priority events in a timely manner. The AWS IoT receives data for supporting business and operational intelligence (refer to Figure 2-10). The lambda architecture is a perfect match, as it combines batch and real-time processing. The former would be good for business reports, and the latter for supervision and operational control. Custom applications may provide domain-specific user interfaces and synthetized services.

Figure 2-10. The potential architecture of an IoT solution for manufacturing

For an excellent overview about implementing an AWS-based lambda architecture, take a look at the blog post “Unite Real-Time and Batch Analytics Using the Big Data Lambda Architecture, Without Servers!” by Laith Al-Saadoon (Amazon).

Summary

AWS IoT is one of the latest additions to AWS, and allows a multitude of devices to connect to the cloud. This makes AWS services available for all sorts of advanced data analytics, including Big Data technologies. AWS IoT also encompasses a lighter version of itself (called Greengrass) that can be deployed near devices. Edge computing is important to efficiently manage feedback loops, and prevent swamping the network and cloud with useless messages (for example, regular OK status reports from devices). Table 2-4 summarizes the characteristics of AWS IoT. Evidently, much of the traits are simply reused from AWS.

To fully leverage AWS IoT you will need to know about AWS networking. For a good overview, watch the webinar “Networking in Amazon Web Services (AWS)” by Richard A. Jones (Addison-Wesley Professional).

Processing colossal amounts of data requires sophisticated knowledge in the realm of machine learning (a subdomain of artificial intelligence). You will find some interesting ideas on this topic in the video “O’Reilly Artificial Intelligence Conference 2017 - New York, New York” (O’Reilly).