12. Embedded Devices

We might not realize it, but we’re surrounded by embedded computer systems. In this chapter, we take a look at what they are, where they are, why they can be a threat, and what we can do to mitigate the threat.

This brings an important point to mind that we should clarify at the beginning. Embedded systems permeate our lives, but not all of them are a threat. I doubt that you would consider your microwave oven a threat, but most modern microwaves have a microprocessor controlling them. It’s only when we start connecting them to the network that we’re going to start getting worried.

Précis

First, we’re going to discuss what an embedded system is, where they’re located, and why you should be worried. We’ll then go through the initial health check to see what kind of exposure we have. We’re going to use some tools that we’ve used before, namely nmap, to find the embedded systems that are on your network. Then we’re going to go through a methodology that will help you identify what your exposure is through your embedded systems. We have some interesting examples to discuss.

We end the chapter with a discussion of the applications, networking issues, and some tools that may help us with the problem.

Special Points of Interest

Embedded systems exist everywhere around us. They’re in our peripherals, our appliances, our medical equipment, our avionics, and our vending machines. We literally can’t turn around without running into an embedded system.

Embedded systems can be carriers and spreaders of digital disease and not even know it. Just as our desktop systems are networked, designers of embedded systems understand the benefit of networking them, and therein lies the foundation of the problem: The networks that make these systems easier to manage and more useful are the same networks used to attack us.

Although we might think of most embedded systems as outside of our sphere of worry, a direct link exists between industrial control systems and our data networks through Ethernet. This opens up a whole new world of exposure and an entire galaxy of threat agents that we’ve never considered.

What Is an Embedded System?

An embedded system can easily be defined as any purpose-built computing system. The first modern embedded system is generally agreed to have been the Apollo Guidance Computer (AGC) developed by the MIT Instrumentation Laboratory under the guidance of Charles Start Draper and Eldo C. Hall. All this little jewel had to do was take navigation information from various sensors and convert it into the outputs that controlled the thrusters on both the command module and the lunar modules that went to the moon. Using 4K of memory and a 1MHz clock, the AGC reliably sent men to the moon and returned them back without ingesting a virus or Trojan horse. However, that was way back in the late 1960s. Although state of the art in its day, that system wouldn’t even qualify as a decent printer buffer today.

Generally, embedded systems are designed to operate for years without user intervention or maintenance. Many embedded systems have failure recovery built in to them so that in the event of a power outage or other type of failure, they can self-recover. Self-recovery proves pretty handy if the embedded system we’re talking about is the one controlling an elevator or rail shuttle.

Another characteristic of embedded systems is that they’re usually, and I say “usually,” based on real-time operating systems such as QNX, a POSIX compliant real-time operating system.¹

Where Are Embedded Systems?

The first point of interest is that embedded systems are everywhere! We tend to identify with the computers that we see on our desk or on our laps because they offer an easy way to identify them: They have a display. However, we have an “out of sight, out of mind” kind of attitude regarding the other systems that don’t have an obvious computer display. Because we don’t “see” them, we tend to forget that they’re even there. For example, the automated teller machine (ATM) is an example of an embedded system that you run across every day.

Another example of an embedded system is your printer, as shown in Figure 12-1. Your printer most likely has an embedded controller in it that manages data coming from the Ethernet, provides a Web-based interface, and manages the print engine. If your printer is a network printer, it does have an embedded controller because it will also have a Web interface on it for remote management. Checking your printers is easy; just try to browse to your printer’s address! If you get a Web page, there’s an embedded system in your printer.

Figure 12-1. A multifunction printer with a Web interface is a good example of an embedded system. This one happens to be the author’s trusty HP OfficeJet 7130 that we pick on in this chapter.

A popular embedded system is the Apple iPod. Essentially an Advanced RISC Machine (ARM) processor with dedicated silicon for the audio encoders and the display, the iPod is a very capable system indeed. Yes, there’s more packed into an iPod than some encoders and a display, and some folks are taking advantage of that by using iPods as bootable drives.² Some really enterprising folks are actually porting Linux³ to the iPod!

Not all embedded systems are as easily recognizable. For example, the computer that controls the engine in your car is a purpose-built closed-loop process control computer (see Figure 12-2). It takes analog signals from various sensors, such as the oxygen sensor, and converts them into digital data that can be used to control the spark timing and fuel-to-oxygen ratio. The Engine Control Unit (ECU) also manages a communications bus that uses a standard called Controller-Area Networking (CAN). Information such as diagnostics is pumped through this bus so that the “auto technician” can troubleshoot engine problems.

Figure 12-2. ECU block diagram showing analog output to cruise control, analog inputs for nitrous oxide and oxygen, and digital outputs to fuel injectors, spark plugs, and fan. ECU communicates over CAN to the antilock braking system.

This discussion takes us right into the area of industrial controls. CAN is a physical medium that allows for high-level protocols to ride on top of it in a way that is similar to how Ethernet works. In the industrial world, CAN is one of the physical mediums that supports the Common Industrial Protocol (CIP). A recent change to the industrial floor has been the introduction of CIP over Ethernet.⁴ Yes, an industrial control protocol over our beloved Ethernet. Why? Because of the TCP/IP stack and the versatility and standardization that the TCP/IP stack brings to industrial control engineers. Ethernet is ubiquitous, and the physical interface chips are cheap. Let’s not mention that CAN is a 500Kbps protocol and the Ethernet hardware is easily an order of magnitude faster. Instead of worrying about getting the message between systems, engineers can concentrate on enriching data structures and making systems more efficient.

Why Should I Worry?

Because they are buried and you might not know where they are, and not knowing that something is out there waiting for you to make a mistake is very bad. Ask anyone who doesn’t have a RADAR detector.

Another reason to be worried is called SCADA. Supervisory Control and Data Acquisition systems are the computers that watch the computers that control industrial robots, pumps in sewage-treatment plants, and control rods in nuclear reactors. For years, these systems talked to their programmable logic controllers (PLCs) and remote terminal units (RTUs) over serial lines such as RS-232 and phone lines. A communications protocol called IEC 60870-5 specifies how packets will be formulated in much the same way that Simple Mail Transport Protocol (SMTP) specifies how email is moved around. For example, in Figure 12-3, our control computers talk to the PLCs via a serial line. The PLCs talk to their sensors through a dedicated channel. Think of PLCs as big honking control computers. They do essentially the same thing that the ECU-type embedded systems do, except that they do it on rail systems, in power stations, and in chemical plants.

Figure 12-3. A SCADA display as one might see in a railway control room. Note the connection between C2 and PLC1 and that PLC2 is down (as is a rail sensor—black dot).

Getting back to why you should be worried, in the old days locomotive systems and the power grid were separated from other networks for two reasons:

1. Specialized control functions meant specialized protocols.
2. Serial connections didn’t provide long-distance links.

In today’s control networks, it’s possible to have the link between C1 and the PLCs be a network connection rather than a dedicated serial line or a modem connection. In that case, Figure 3 might not accurately indicate all the systems that have connectivity to the PLCs.

The problems of effective utilization of equipment, and in some cases reusing of old equipment, has forced SCADA implementers to turn to more effective communications protocols: in this case, Ethernet. A number of vendors provide bridges between Ethernet and RS-232, and these vendors are actually touting the capability of their products to be remotely managed via TELNET, HyperText Transfer Protocol (HTTP), and Simple Network Management Protocol (SNMP). One vendor, Data Comm for Business, is even bragging that their system can be used over Cellular Digital Packet Data (CDPD) networks!⁵

So, just to sum up, we’re linking serial and Ethernet and tossing in the cellular network, and at the end of these networks are endpoints that control critical infrastructure systems.

Another reason for concern is that embedded systems are showing up in places where you might not expect (such as medical equipment). Not that embedded systems in medical equipment are bad, but newer medical equipment is coming equipped with network interfaces, and that has the potential to be very bad.

None of this would be especially bad if the embedded industry kept up with the rest of the industry. The embedded industry is fairly entrenched and seems to be about five years behind the rest of us. They’ve recently added networking capability (and, as you’ll see later, not in a secure way).

Embedded Threats

We’ve covered a number of threats in the Windows, Mac OS X, and handheld systems; now we’re going to discuss a few of the vulnerabilities that afflict their embedded cousins.

QNX Software Systems’ Neutrino RTOS (Real-Time Operating System) has, like all other operating systems, vulnerabilities posted by organizations such as iDefense.⁶ Although discovered by an individual in 2004, iDefense reports that public disclosure occurred in February 2006. According to iDefense, the vulnerability is in how a program called phfont spawns another program called phfontf.⁷ An attacker can replace phfontf with something evil, and because phfont spawns phfontf, the attacker can set UID root, giving the evil program root privileges. Just the kind of vulnerability I want in my nuclear reactor controller. The good news is that iDefense reports that QNX Software Systems hasn’t even acknowledged that the vulnerability exists. I followed up with some checking and discovered that like other operating systems, QNX has a history of vulnerabilities that can be remotely exploited by a malicious attacker.

You can find a fairly complete list of current vulnerabilities at www.openqnx.com/index.php?name=News&file=article&sid=417.

The nine vulnerabilities on this list can result in root access. The only “fix” so far is to grant trusted users only access to the QNX-equipped systems.

Not enough? How about the fact that some Digital Subscriber Line (DSL) modems run embedded systems that allows a remote attacker to gain access via the Web server? I talk more about this in the following section.

There has been considerable talk in the recent past about how the JetSpeed 500, JetSpeed 520, and Allied Data Technologies CopperJet 811 RouterPlus DSL modems allow an attacker to gain access to these devices via the embedded Web server provided by Virata-EmWeb.⁸ I did some digging and found that the same Web server, Virata-EmWeb/R6_0_3, might be responsible for a vulnerability in the 3Com SuperStack 3 NBX and 3Com NBX 100 network Voice over IP (VoIP) solutions⁹ that can result in a denial-of-service attack.

To make things a bit more personal, embedded systems are creeping into your house and on to your home network. These systems will allow an attacker to control the locks on your front door and your thermostat, all from the comfort of the Internet. Some folks are using Web cams to monitor their babies, not realizing that the device is a conduit for attack.

Initial Health Check

Once again, the initial health check is more a case of being able to identify what your exposure is, because you’re not going to be able to do much about it. There are two ways to approach this problem: blind scanning and pinpoint identification.

I’ve been through both processes, and any way you stack it, it means work. Asset management systems claim to be an answer, and although they will clearly help in situations in which some form of provisioning system exists, they rely on humans to enter the data and are therefore inaccurate. You must go through the scanning process to determine how much change there has been and what that change is.

Blind scanning is simple: Run a scanner such as nmap, pop in your IP address range, wait for the results, and pump them into a spreadsheet or database. You can then map that to your known list of desktops, servers, notebooks, routers, switches, printers, and other addressable bits that you have in your inventory. The things that you don’t have on the list will stick out like a sore thumb. This will take some work, and you must write some scripts to make it work. Nmap has a feature that enables it to save machine-readable logs, thereby making this easier. If you have a large, distributed network, this is probably your only real choice, and it makes some pretty broad assumptions. First, it assumes that you have an inventory that you can compare things to. Second, it assumes that you have complete access to the entire network.

A word of warning on scanning, however: There has been some discussion about the inability of some SCADA systems to endure a network scan. Some systems are fragile and might need some attention. You might even have to hunt down dead systems.

The other method works for small shops and involves identifying potential target devices. The scenario runs something like this: You walk into a cube and see a new printer that wasn’t there before. You can select through the printer’s display menu to the network section. It doesn’t make a difference what printer it is; most of them have fairly intuitive menus. When in the network section, look for the IP address section and copy that address down and return to your scanner.

I ran the scanner against my printer just to see what I could find out. As you can see in Figure 12-4, the printer is running and providing a complete TELNET interface to the network!

Figure 12-4. A screen capture of an nmap scan of an HP OfficeJet 7130 All-In-One printer, scanner, and fax. Note that port 23 is open and waiting for a TELNET connection.

Finding that TELNET was running on the printer, I did what any self-respecting geek would do: I started a session! Figure 12-5 shows the results. As you can see, a complete menu is offered over the network. You can manage which network protocols are offered, how the printer is configured, and get help. Who needs windows!?

Figure 12-5. The menu provided to the network by the printer’s management interface.

I navigated through the TCP/IP settings to the Access Control screen, as shown in Figure 12-6. From this menu, you can configure how the printer responds over TCP/IP. You can set ports and print options, but the interesting selection is, of course, #4, “Access Control.”

Figure 12-6. The TCP/IP settings menu allows the administrator, or a hacker, access to how the printer behaves on the network.

Entering 4 and pressing Return brought me to the screen in Figure 12-7. What this figure displays is that a primary security mechanism for this printer is IP address filtration. All I had to do was to manually enter the IP address that the filter allowed into another notebook, and I was allowed to access the printer.

Figure 12-7. Simple IP address filters are the first line of defense in this embedded application.

If we explore the menus on this printer, we also discover that it supports SNMP. SNMP enables network administrators to query the printer for status or set specific parameters. The HP printer that we’re looking at has SNMP enabled, as you can see in Figure 12-8.

Figure 12-8. An unset SNMP community string is inviting unwanted access. This endpoint allows SNMP read and write access by default.

With the Cmnty Name (short for community string) unspecified, this printer has the capability to write to other devices on the network to store things such as scanned images, pictures from cameras, or faxes. That means that if I can control the printer, and the printer has a trust relationship with other endpoints, I can write data to them at will.

Because the printer was accepting packets on port 80, I decided to try to TELNET to it to determine what Web server was running. I entered the command shown in Figure 12-9, but was greeted with only the standard escape character message, as shown. However, when I entered a bad request, the Web server puked, telling me that my request was bad (HTTP/1.1 400 Bad Request) and, to help me troubleshoot my error, the type and version of the Web server were provided. This is what put me down the path investigating the Virata-EmWeb vulnerability discussed earlier.

Figure 12-9. Error messages can reveal information about a target endpoint (in this case, what Web server is running on the embedded system).

None of this was rocket science. I think I invested about 15 minutes going through the entire process, and I didn’t do anything special. All I did was this:

1. Scan the network
2. Probe some ports
3. TELNET to some open ports
4. Take control of the endpoint

Now in the interest of fairness I have to tell you that the printer can set a password. As you can see in Figure 12-10, it’s almost a suggestion, and the default is no password.

Figure 12-10. Password protection on this embedded system is not enabled by default.

When you set a password, there are no checks for minimum length. I set the password to “a,” and the system was fine with that. When I did set a password, the system asked for a user name (stating that “root” or “admin” are acceptable user names). The truth is that any user name will work! When asked for a user name, I entered “blaa”; the system then asked for the password. I entered my super-secure password of “a,” and I was back in.

The point is that if it’s this easy to do, I’m not the only one doing it!

Applications

Normally, when we think of applications, we think of chunks of code that enable us to generate documents, create presentations, or capture and store photos. Generally, anything that runs on our general-purpose computers is an “application.” Some of them are utilities that enable us to configure widgets here and there, and some of them are full-blown suites that enable us to write an email to our significant others. Any way you look at them, they’re specialized code running on a generalized architecture. This is not the case with embedded systems.

Embedded applications are really best described as the how the endpoint is used rather than what’s running on it. In some cases, the embedded system is managing the power plant in your car, or in some really benign cases, managing the CD that’s presently in the player. In the industrial applications, embedded systems are running the following:

• Assembly line robots (painting and welding)

• Locomotive systems (trains and tracks)

• Electrical power grid (load controllers)

Moving from the industrial to the medical, we’ll find embedded controllers in the following:

• Patient care systems

• ECG/EKG (Electro CardioGraph)

• Ventilators

In your office or home, you’ll find embedded controllers in these:

• Handheld devices

• Printers

• Scanners

• Vending machines

• VoIP handsets

Just to ram this point home, Samsung and LG have home network solutions that connect things such as your door locks, washing machine, and gas valves to control pads and your home PC. The Samsung page has this cute animated graphic that shows a path from the Internet to your gas valve!¹⁰ It gives new meaning to the notion of denial of service.

So, we need to change our idea of what a computer looks like, because it can, and has for some time, looked like something other than a PC.

Networking

In an interesting fusion of technologies, I found an embedded controller being sold by a company called VDI Tele Assist. Called the CardioGSM, it records and transmits heart information to the hospital or patient care facility. If you thought that the GSM part looked familiar, you would be right. The CardioGSM uses the cellular network as a data conduit. The CardioGSM also functions as a cell phone so that patients can chat with the doctor while placing the 12 leads required to gather data on their bodies. I want to point out here that the cellular network is a full-duplex, bidirectional, insecure network. I think that my heart rate might elevate a bit if I were to attach such a device to my body!

As discussed earlier, Controller-Area Networking provides an internal type of communication medium that enables the actuators and sensors to share acquired information and control signals.

The IEC 60870-5 protocols are designed to allow the exchange of telecontrol messages between two systems and as such is really a circuit-based protocol in much the same way that DNP3 is.¹¹ DNP3 is designed to connect a SCADA master to RTUs that control devices such as PLCs and other “smart” control devices. The main reason that these protocols exist is to provide some level of interoperability between equipment in power system control solutions. These protocols become more interesting when they’re transmitted over Ethernet and the IP protocol suite.

Tools and Vendors

This section is usually reserved for vendors and tools that can help you solve the problem discussed thus far in each chapter. Unfortunately, the only vendor that was doing security for embedded systems was a company called Sygate Technologies, and they were bought by Symantec in late 2005. The unfortunate aspect of any acquisition is the inevitable delays and cancellations that result from “realigning” and “integrating” the acquired company. It is unclear at this time what the future holds for Sygate’s technology.

The remaining companies consider security to be the implementation of a virtual private network (VPN). SafeNet offers a product that is aimed at printer vendors and offers a fairly rich set of VPN tools, but that’s pretty much it.

Certicom seems to have targeted VoIP, but it also supports traditional endpoint security products.

Embedded Security

That leaves us with the work that the TCG is doing. Specifically, the Trusted Platform Module (TPM) is a hardware device that must be embedded within the hardware architecture before it can provide any services.¹² You might be thinking “smart card,” and you are close. The TPM is in itself a microcontroller that stores keys, certificates, and hash values in a tamper-resistant environment. Because it is a microcontroller, you can also use the TPM to generate keys dynamically. However, it must work in conjunction with a processor.

Many of the TCG’s specifications are based on the trust foundation provided by the TPM. However, the TPM can’t work alone and must rely on external trust agents to provide checks. For example, when a system boots, part of the kernel checks the hash values of modules as they load. The kernel checks with the TPM to verify that the hash values are correct. The Trusted Network Connect (TNC) has been referenced in this book as being an open standard that a CLPC solution can be based on. Once again, I want to point out that the TPM alone will not accomplish the tasks required of a true CLPC solution. I also want to point out that there are many hurdles to implementing the TNC as envisioned by the TCG. Although the TNC specification does discuss obtaining some measure of integrity via the Integrity Measurement Controller (IF-IMC), the discussions are too high level to be of any real use.

It’s interesting to note that all the major manufacturers of PCs have jumped on the TPM bandwagon. You can purchase a PC with an embedded TPM from the following:

• Lenovo (was IBM)

• HP

• Gateway

• Fujitsu

• Dell

• Toshiba

• NEC

• Sony

Missing from this list are embedded system designers such as Wurlitzer. Yes, Wurlitzer. Deutsche-Wurlitzer makes vending machines, and one of the options is a cashless payment system. In addition, Deutsche-Wurlitzer has been working with VendSMARTT through a partnership with SL-Tech¹³ to create Healthy Vending at local schools. The vending machines are interfaced with the SL-Tech food service computing systems via the network, both wired and wireless.

Closing the Loop

Although embedded systems, PLCs, and SCADA systems are dedicated control-based devices and take advantage of many internal closed-loop processes, they are light years behind even the mobile development community. Network access control isn’t even considered as part of the roadmap at this time.

Basic security functionality, such as vulnerability management and antivirus (AV), has to be addressed before we can begin to discuss the advanced concept of trust-based CLPC within the context of embedded systems. Combine that with the push to add network functionality to SCADA systems, and we have a recipe for disaster.

Key Points

Embedded system are all around us, and for some reason we have ignored the security features that should be embedded within them. We have no real security policy that addresses them, nor do we seem to be pushing vendors to do anything about it.

We’re Surrounded

We are all quite literally surrounded by embedded systems, and the number is growing each and every day. Medical equipment, industrial controllers, and vending machines are all around us. Video cameras provide Web interfaces that make it easy to integrate them with your Web applications. Our traffic light system is essentially an embedded system, and we trust it with our lives. Have you ever given thought to what would happen if all the traffic lights were green at the same time?

In a very short time, most of the phones that we’re going to rely on will be VoIP powered and therefore will all contain some form of embedded controller in them. I remember a time when I could pick up a phone in the middle of a power outage and still get a dial tone. Not so with the complicated answering machine/walkie-talkie/multiple-handset phones that we’re buying now. When the power goes out, so does the phone. For personal security, I always keep a simple old analog phone connected so that in case the aliens come and nuke our power, I’ll still be able to call someone to complain about it.

Medical equipment is relying on embedded controllers to acquire, store, and transmit patient care information. In some cases, our very lives are in the hands of embedded systems. When we drive our car, we’re relying on numerous embedded systems, such as antilock brakes, to keep us safe. BMW even has a model with adaptive cruise control that keeps you from slamming into the car in front of you when the moron is only doing 55mph in a 65mph lane.

No Real Security

Unlike the tidy world of Internet security, there is no single vendor of embedded security solutions. Although the TCG does have the TPM, and it is a piece of hardware, it’s not a piece of hardware that’s making its way into our everyday home network controllers, such as those offered by Samsung and LG.

No standard says how embedded systems are to be tested or just what services should be offered. Although I whine about the fact that vendors have been driving our security solutions, the embedded industry has nobody driving the boat.

There’s no concept of embedded firewalls unless you’re talking about a DSL modem. Then a firewall is a “feature.” Even our DSL modems are turning out to be vectors of attack, however, because of their embedded systems. Worse, with the multiple ways that we can connect to embedded systems and controllers, we’re creating hidden paths that will allow an attacker to sneak into our networks not by outsmarting the security, but by going around the security.

TPM Isn’t Helping Embedded Solutions

Embedded systems are the classic cost versus security problem. Why? Because one of the stated objectives of using an embedded system is to engineer it to be specific to the task that it is trying to accomplish. By being specific, you reduce cost and improve performance. We’re in a world now in which we’re talking about $100 per “seat” of security tools on every system on our network. Increases of $7 to $10 a seat receive minute scrutiny. I’ll tell you right now that security isn’t going to fly in a world where things have to cost less than $5 a piece to make. Adding TPM to your $5 solution just doubled the cost to $10.

Embedded systems are also at a different stage in their development as networking systems than our enterprises are. Embedded systems haven’t had to endure the years of relentless attack that Windows, and our networks, have had to suffer through. As more embedded systems come online, however, that will change.

Closing the Loop

There is no easy way to say this. Embedded systems, PLCs, and SCADA systems represent the largest threat to our security that I’ve seen in a long time. Not only is the loop not closed, but in some places the loop isn’t even being discussed. SCADA systems control critical infrastructure, and they do so with obsolete and vulnerable technology.

Basic security functionality must be added to these devices in the next generation of products if we are going consider CLPC security technology in the near future.

You Can Do Something

We do have some tools that we can use to identify embedded systems on our network. After we identify an embedded system, we can examine it to ensure that it’s as secure as we can make it, replace it if warranted, or remove it if need be. Most embedded systems are going to have similar security capabilities to our printer example (your DSL modem, for example), but it will take time to ferret them out and configure them properly. Then it becomes a matter of documenting what you’ve done so that when the time comes to change the embedded endpoint’s configuration, you, or your replacement, can do so effectively and securely.