Working with Law Enforcement on SCADA Incidents
Lester J. Johnson Jr. is employed by the SCANA Corporation, a $ 9 Billion, Fortune 500, energy–based holding company, headquartered in Columbia, South Carolina. Mr. Johnson serves in the Corporate Security and Claims Department as a Manager with responsibility for Investigations and Crisis Management. Mr. Johnson leads a staff of professional investigators who conduct investigations of internal corporate compliance issues, criminal violations against the corporation’s property and personnel, executive protection, background investigations and risk reduction efforts on behalf of the Corporation. The Crisis Management Department is responsible for the development and continual assessment of security risk management and reduction plans for the critical infrastructure operated by the Corporation. These risk management and reduction plans include the assurance of compliance with the various governmental agencies with oversight responsibilities for the critical infrastructure. Business continuity and emergency procedure planning are also a major component of the crisis management group.
Mr. Johnson is a retired Deputy to the Assistant Director of Investigative Services for the South Carolina Law Enforcement Division. He was responsible for the delivery of all investigative services, which included general investigations, bomb and arson, tactical, computer crimes, special victims, executive protection, behavioral science, public corruption, and insurance fraud.
During his twenty–eight year career Mr. Johnson received numerous awards and commendations, including the Strom Thurmond Award of Excellence in 2004. Mr. Johnson is a graduate of the FBI National Academy and has served as an adjunct instructor for numerous organizations.
Mr. Johnson is married to the former Laura Whelchel of Cordele, Georgia and resides in Lexington, South Carolina with his two children.
Introduction
Law enforcement agencies are receiving increasing requests for the investigation of high technology related criminal activities. The protection of the nation’s critical infrastructure is among the highest level priorities for ensured continuity of operation for our nation. At the core of many of these critical infrastructure operations are systems that move data to and from the computerized system that manage and control the operations of the infrastructure. As you can imagine, should these systems come under attack or become compromised and control is assumed by those person(s) intent on creating havoc, a disastrous outcome is imminent. The potential exists for great harm to human life and the national economy if SCADA systems are breached. For this purpose, it is my intent to, in many cases, introduce SCADA systems to law enforcement and discuss potential responses to investigating an attempted or actual breech of these systems. Additionally, you will find several examples of mitigation that law enforcement can pass along in a pro-active attempt to prevent occurrences of system breaches.
The SCADA systems described below are very technical in nature. Law enforcement agencies who lack the personnel with a strong information technology background will find it very difficult to grasp the concepts associated with SCADA systems. Having the knowledge of the system and the associated audit logs are necessary in determining the point of an actual or attempted breach and discovering and preserving the evidence necessary to support criminal charges.
SCADA System Overview
SCADA stands for Supervisory Control and Data Acquisition. SCADA refers to a system that collects data from various sensors that send data to a central computer, which then manages and controls the systems. There are many parts of a working SCADA system. These systems usually include signal hardware input and output, controllers, networks, human machine interface (HMI), communications equipment, and software. The term SCADA refers to the entire control system. The control system monitors data from various sensors that may be located in one facility, or they could be geographically distributed across a city, state, or even around the globe.
These SCADA systems are core to much of the nation’s critical information technology infrastructure, most of which is owned and operated by private industry. It is vital that both the system owner and law enforcement be proactive in developing partnerships, building collaborative capabilities, and defining processes to facilitate investigations in the event of an attack.
Let’s spend just a minute to explain the basic components of a SCADA system and its functions. This will help law enforcement understand the possible attack vectors as we go through the details of each and how to protect them.
One of the key aspects of a SCADA system is its ability to monitor an entire system in real time. To accomplish this, SCADA systems incorporate remote terminal units, or RTU’s, and programmable logic controllers, or PLC’s. These RTU’s and PLCs are the intelligent building blocks of a SCADA system. Usually RTU’s or PLC’s run a pre-programmed process, such as turning a switch on or off, opening or closing a valve, or sensing temperature changes. Monitoring each of them individually can be difficult, because they often are spread out over the system. The human machine interface HMI is where the information is displayed or monitored by the operator. HMI’s are an easy way to standardize the monitoring of multiple RTU’s or PLC’s. Real-time communications to these units is critical to the health of the system. Because of the possible link to physical outcomes, the requirement for redundant systems is heightened. HMI’s often are connected to databases to facilitate providing graphs or trends, logistic information, schematics for a specific sensor or machine, or even troubleshooting guides. SCADA systems are considered closed loop systems and run with relatively little human intervention.
An attacker who wishes to assume control of a SCADA system is faced with three challenges:
Unfortunately there are many methods an intruder can use to get this information. Increasingly these systems are well documented on the web and are easily accessible.
Understanding the control systems cyber vulnerabilities is key to understanding the steps needed to protect these systems. The British Columbia Institute of Technology (BCIT) study of 34 incidents shows the SCADA components breached and their origins.
The following charts are divided into two groups, 14 internal events and 25 external. Notice the Virtual Private Networks (VPN); those trusted networks lead to 8 percent of the breaches. Given these statistics, one can begin to get an idea as to where one should prioritize the spending of security money.
The BCIT’s report says 70 percent of attacks reported between 2001 and 2003 came from some external entity. Between 1982 and 2000, just 31 percent of attacks came from the outside. This is an older study, but the trends have continued.
The sharp increase of SCADA intrusions beginning in 2001 indicates a sharp increase in these kinds of attacks, which may or may not be politically motivated.
Secure Network Management
Securing Wide Area Network Perimeter
A well-segregated network is your first step to controlling your systems. The first segregation point is a perimeter security defense using a firewall between the business network and the Internet. Utilize an Intrusion Protection Devise (IPS) as well to monitor and automatically block unwanted traffic entering from the Internet. Do not forget your wireless access points. Treat them the same as an Internet access. Use a firewall to segregate your SCADA control LAN from your business network. Use an IPS to monitor and block unwanted traffic from your business network to your control LAN.
Controlling Access
Access control from a physical perspective is obviously important. Secondly and statistically more important is network access control. A human actor using network access can wreak havoc on an unprotected or poorly managed network. Once access is gained, control is just a step away. Using technology such as two-factor authentication is strongly recommended. Tokens, swipe cards, and even biometrics in conjunction with a strong user id / password policy go a long way to controlling access. Once you know with whom you are dealing, then controlling what he or she does is your next challenge.
Performing Network Backup and Recovery
The backup and recovery of a SCADA system has its own unique issues. The all-the-time requirements of SCADA systems means that backups must be done on live systems and recovery times must be very short. The solution is often a mirrored database that can be taken off-line for backup purposes without affecting the production systems. Otherwise business production lines would be shut down, possibly resulting in loss of money and business reputation. Use as much caution as to who and how access is given to these systems as you would the control system itself. The data in these databases can be hacked, giving away enough information to fully understand the processes needed.
Transmitting Legacy Non-Routable Protocol Securely
Many of the SCADA systems have been around long before the advent of the PC or Internet, and many of their communication protocols were developed for closed networks. Transmitting these protocols over a routed network was never envisioned. The Internet protocol Ethernet is designed to get the message there through whatever paths are available and is not a guaranteed delivery protocol. Many of the SCADA systems are very time sensitive and must be delivered on time every time, so using alternate routes that take more or even less time can cause issues.
Dial-Up Access to the Remote Terminal Units (RTU)
Modems are often used as backup communications if the primary high-speed lines are down. The attacker will automatically dial every phone number (known as WAR dialing) or every extension in the company looking for modems hung off the corporate phone system. Most Remote Terminal Units (RTUs) identify themselves and the vendor who made them. Most RTUs require no authentication or a password for authentication. It is common to find RTUs with the default passwords still enabled in the field.
Vendor Support: Dial-Up Modem/ VPN Access
Most control systems come with vendor support agreements. There is a need for support during upgrades or when a system is malfunctioning. The most common means of vendor support used to be through a dial-up modem and PCAnywhere. In recent years, this means of support has transitioned to VPN access to the control system LAN. An attacker will attempt to gain access to internal vendor resources or field laptops and piggyback on the connection into the control system LAN. Monitor and manage this access point closely for possible attacks.
IT Controlled Communication Gear
Often the routers and telephone communications are maintained by the business IT staff. A skilled attacker can compromise these devices by gaining access to the field communications.
Corporate VPNs
Virtual Private Networks (VPNs) are often set up for access to the control networks from the business LAN. An often-used attack vector is the end user’s workstation utilizing key-loggers and screen scrappers, which piggyback on the legitimate users as they access the control systems through the VPN’s. Extra care should be given to every workstation that is allowed to access the control network. Ensure up to date anti-virus software, patch the operating system automatically, and make sure that host-based firewalls are standard on these workstations. Consider whether to allow workstations that access the Internet to connect to the control LAN. Strong cyber security user education training should be required. The business LAN is the most likely access vehicle attackers will attempt to use to access your control systems. The e-mail, web servers, and users browsing the Internet are the most likely entry mechanisms. The latest operating system patches and up-to-date anti-virus software are a must. In 2007 there were, on average, 19 new exploits per day, with an average of a 30-day lead-time between known vulnerability and patch availability. Thus, on any given day, there are possibly 570 ways they can get into your systems that you have no way of stopping. Security in-depth is your only defense.
Database Links
Databases are often mirrored to the business LAN for disaster recovery or backup purposes. They contain the keys to the entire control systems, and include user id’s and passwords, control codes, detailed explanations of how the control systems work, and information about the RTU’s and PLS’s that control each component of the system. Attackers use specially crafted Sequel Query Language (SQL) statements to take over the database server and extract its contents. Every modern day SQL database allows this type of attack if steps are not taken to block these statements.
Poorly Configured Firewalls
Not providing outbound data rules is the most common configuration problem. This may allow an attacker who can sneak a payload onto a control system machine to call back out of the control system. This is known as phoning home. This is one of the most widely used techniques, often taking advantage of vulnerability in a browser and subsequently pulling down malware from another website without any user interaction.
Business Partner Links
These links have often been thought of as trusted relationships, but what it means is that your systems are only as well protected as your weakest link. The business partners are not automatically that weak link, but it does increase you exposure and make the process much more complicated to secure. Many times, dictating to a vendor how its networks are run can be difficult and verifying that the vendor is following your requirements is even harder.
Managing Security Events
Utilizing a Security Event Management System is the key to effectively managing critical systems in today’s Internet-connected world. Without eyes on target giving you situation awareness, you really don’t know what is happening on your network. What you don’t know can cause you big problems. Most networks use IDS/IPS devices to monitor the front doors and access points to the Internet. There are no more perimeters in today’s networks. Devices like PDAs and cell phones with Internet access are inside your building and behind your firewalls. Your partners and vendors are on a virtual private network connected into your network from half a world away. Insider attacks constitute the majority of today’s attacks. The SEMS can be used in a post-mortem fashion to assist in lessons learned or during an investigation to determine how the attack was carried out.
Once your devices are centrally logging, a correlation engine that brings all the events together and visually displays the events in real-time is vital. Filtering through thousands of events daily without this capability would be impossible. The objective is to get out in front and proactively identify attacks before they are successful. Attackers must go through a discovery process or finger printing of your systems prior to penetration. The visibility that you have with a SEMS will afford you the knowledge needed to stop these attacks prior to their gaining a foothold on your network.
Conduct Routine Assessments
A security assessment shows the system’s security posture with respect to specific vulnerabilities. The assessment closely examines the network architecture and interprets and compares results against the various business processes to determine whether security controls are in place to appropriately address the vulnerabilities. The assessment testing methodology must be effective at exposing vulnerabilities on complete systems, including networks, applications, operating systems, wireless, security devices, and all supporting systems and their many interconnections. There are many automated tools that can be used during these assessments. Social engineering is also a vital component of an assessment. Remember its people, processes, and technology. Technology is never the complete answer, and often the people and processes are overlooked in these assessments.
Examples of Common Attack Techniques
Man-In-The-Middle Attacks (MITM)
An MITM attack is when the attacker makes independent connections with the victims and relays messages between them, making them believe that they are talking directly to each other when in fact the entire conversation is controlled by the attacker. The attacker must be able to intercept all messages going between the two victims and inject new ones, which is straightforward in many circumstances. For example, the owner of a public wireless access point can in principle conduct MITM attacks on the users.
Key-Logger Software
A key-logger is a type of surveillance software that records every keystroke you make to a log file, usually encrypted. A key-logger can record instant messages, e-mail, and any information you type or that is displayed on your monitor. The log file created by the key-logger then can be sent to a specified receiver. Some key-logger programs will also record any e-mail addresses you use and Web sites you visit.
Summary
Given on average 570 exploits a day, no way to patch the operating systems, and 450,000 infected pages, how can you ever keep your systems protected?
Situational awareness, segregation, security event management, security assessments, and security training are the points that should be stressed.
1 Know what traffic is on your network. Know it well enough that if your traffic goes up or down by 10 percent to 15 percent you know it and you can tell exactly why.
2 Keep your security zones intact. Never access a higher security zone network or device from a lower level security zone or device.
3 Centrally log and correlate all your devices from the core to the perimeter. Correlation between known system vulnerability and attempted exploits against those systems is your actionable data. Actionable data in a sea of otherwise meaningless data is pure gold.
4 Trust but verify with routine assessments of your people, processes, and technology. Scanning or other automated assessments are very beneficial but the people and the processes are where you will either pass or fail on a day-to-day security test. Absolutely do your automated testing of operating systems, virus-patch levels, databases, applications, and network devices and do them often, quarterly or more frequently on the higher value targets. At the very least, yearly review your processes, audit your security training effectiveness, and perform some social engineering audits.
5 Training your IT staff on the possibilities and your users on the impact of their actions can have a very positive affect on your security. Statistically, well-trained people and properly followed processes are the keys to a successful security program, yet we find over and over the lack of training and poorly followed processes.
Solutions Fast Track
SCADA System Overview
SCADA systems are core to much of the nation’s critical information technology infrastructure.
It is vital that both the system owner and law enforcement be proactive in developing partnerships.
One of the key aspects of a SCADA system is its ability to monitor an entire system in real time.
Understanding the control systems cyber vulnerabilities is key to understanding the steps needed to protect these systems.
Secure Network Management
A well-segregated network is your first step to controlling your systems.
A human actor using network access can wreak havoc on an unprotected or poorly managed network.
Backups must be done on live systems and recovery times must be very short.
Many of the SCADA systems are very time sensitive and must be delivered on time every time, so using alternate routes that take more or even less time can cause issues.
Where possible, configure modems in the call back mode. This helps with the WAR dialing from being successful.
An attacker will attempt to gain access to internal vendor resources or field laptops and piggyback on the connection into the control system LAN. Monitor and manage this access point closely for possible attacks.
Often the routers and telephone communications are maintained by the business IT staff. A skilled attacker can compromise these devices by gaining access to the field communications.
Extra care should be given to every workstation that is allowed to access the control network.
Your systems are only as well protected as your weakest link.
Managing Security Events
Utilizing a Security Event Management System is the key to effectively managing critical systems in today’s Internet-connected world.
Get out in front and proactively identify attacks before they are successful.
A security assessment shows the system’s security posture with respect to specific vulnerabilities.
Examples of Common Attack Techniques
An MITM attack is when the attacker makes independent connections with the victims and relays messages between them, making them believe that they are talking directly to each other when in fact the entire conversation is controlled by the attacker.
A key-logger is a type of surveillance software that records every keystroke you make to a log file, usually encrypted.
Frequently Asked Questions
Q What are some examples of industry using SCADA systems?
A SCADA systems are used in a variety of plants and processes, from optimizing the most benefit from production lines to supervising and controlling the production of toxic and dangerous chemicals. Some examples are nuclear power generation, electric power generation and distribution, and water purification systems.
Q What are the key elements to secure a SCADA system?
A Situational awareness, segregation, security event management, security assessments, and security training.
Q Between public and private entities, which one is the great user of SCADA systems?
A The private sector is a greater use of SCADA systems.
Q What resources are available to law enforcement for assistance in investigating breaches of SCADA systems?
A Local law enforcement can seek assistance from federal law enforcement, law enforcement agencies that have experienced high technology crime investigative units, or subject matter experts from the private sector.