7
After the Breach: Improve Incident Response across Business Functions
Companies can protect their critical assets appropriately, they can ensure that cybersecurity is embedded in their culture and systems, and they can step up their defense mechanisms. They can take all the actions we have outlined up to now. But they will still be attacked. Cyber-attacks are becoming increasingly sophisticated, more frequent, and their consequences more dire. When determined adversaries set their mind to finding a way inside, every organization with valuable digitized information is at risk of having its critical assets compromised.
Breaches unsurprisingly affect shareholder confidence as investors get spooked by the transgression and often by the slow response of the company to deal with it or even acknowledge it. This impact will be disproportionately heavy for companies trying to exploit new digital business models. CISOs and, increasingly, CEOs are realizing that more business value can be destroyed as the result of a poor response to the breach than by the breach itself. Knowing how to respond to a cyber-attack is not a question of having good instincts. It needs to be learned and embedded. Achieving this level of expertise requires developing a robust incident response plan and then—crucially—continuously testing and challenging it through simulations.
Unfortunately, most organizations are not prepared. They are reactive, not proactive. Despite the recognition that breaches will continue to escalate and that they are likely to be attacked, they still do not adequately think through how to respond. They need to know what to do next.
The U.S. Department of Defense (DoD) spends $5 billion a year on cybersecurity,1 more than anyone else, yet it recognizes that its systems are far from impregnable. Indeed, it assumes that its unclassified network will be penetrated and therefore concentrates on how to maintain day-to-day operations in the event of a breach. Like any good military organization, it has plans in place for all manner of contingencies. It is under no illusion that it can spend its way out of trouble or that technology alone will solve the problem. Instead, its mind-set is one of being prepared to respond when the attack happens.
Companies, which spend far less on protection in the first place, would do well to follow the DoD’s lead. There is no such thing as a foolproof system, and the cybersecurity battle is being fought not only at the outer walls but increasingly inside the city gates. Forward-thinking CISOs are therefore adopting an approach of detection, response, and recovery, and companies are being held accountable by the market, customers, and regulators for how they respond to a breach. All three are becoming ever more unforgiving of an unprepared and uncoordinated response.
Tools traditionally used to detect vulnerabilities, such as penetration testing, are now being used to assess security teams’ ability to detect a breach. One leading technology company, for example, conducts penetration testing every three weeks, but its goal is not to identify weak spots, but rather to test how well its security team can find the intrusion and respond appropriately. How many traditional companies can say they do this? Only 55 percent of the organizations we surveyed conducted any kind of systematic penetration testing. Of those, almost all of the programs were designed to find the vulnerabilities rather than test the cybersecurity team’s skill in detecting breaches.
The IT response is only one aspect of reacting to a breach and often the most advanced. For many organizations, the bigger challenges are coordinating the broader response to a security incident and establishing a culture of continuous learning so their responses improve over time.
To address these, organizations need to take actions on three fronts. First, they need to draw up an incident response plan that spans business functions. Second, they need to test the plan continuously and embed it into the organization through disciplined practice and war gaming. Third, they need to conduct post-mortems of real breaches to assess the effectiveness of the plan and feed the results back.
DRAW UP AN INCIDENT RESPONSE PLAN
The primary objective of an incident response (IR) plan is to manage a cybersecurity incident in a way that limits damage, increases the confidence of external stakeholders, and reduces recovery time and costs. It achieves this in three ways: clearer decision making, stronger internal coordination and accountability, and tighter third-party collaboration.
Clarify Decision-Making Responsibilities
An insurer’s core underwriting system was compromised by malware that was stealing sensitive data. The CISO took the unilateral decision to disconnect the system from the broader network without realizing that this system was a critical piece of the business infrastructure. Taking it offline would cost the company $20 million a day in revenue. After he declared his intent, the head of business operations challenged the CISO’s authority to make what amounted to a strategic decision for the company. The two sat at loggerheads not knowing who had the authority to make the call, while data continued to flow out of the compromised system.
By contrast, when the security team at another insurer confirmed that malicious code had infected a core application, there was a consensual decision by management to shut down complete network access. With decision rights already clearly defined, it was relatively straightforward to determine that the risk of continued data loss from this specific application outweighed the associated loss of revenue resulting from downtime. Furthermore, having developed standard procedures for isolating strategic segments of their network, the technology teams followed step-by-step instructions to efficiently quarantine the application.
When an attack happens, managers from across the organization have to make some quick decisions. Identifying as many of those decisions in advance as possible and codifying them in a plan saves time and minimizes loss. But, as we saw in the first example, it is just as important to have clarity over who has the authority to make the decision based on the value at stake. Too often, companies do not anticipate the kinds of decisions that have to be made in the chaotic moments when a breach is discovered.
Strengthen Internal Coordination and Accountability
One institution delayed releasing an important statement to the media about a cyber-attack while executives debated the desired messaging. The delay was due to unclear responsibilities that slowed down the executive group as its members defined a list of external stakeholders, tried to assemble a meeting with internal stakeholders, and debated numerous issues.
Many of these tasks could have been resolved in advance. During the early stages of a data breach crisis at a large retail bank, customer service reps worked from preapproved scripts to handle calls from customers asking about the nature of the breach. Aside from giving a consistent message to the outside world, this also meant that executives and security teams could focus on investigating the data loss without being distracted by having to develop communications on the fly.
However, even having scripts does not guarantee a smooth response. Another financial institution had scripts in place, but the messages they contained had not been coordinated with the regulatory team who were preparing to give regulators messages that differed materially from what service reps were telling customers. Thankfully, managers spotted the inconsistency in time and were spared a potential public relations nightmare.
It is this coordination that gives meaning and structure to the decision making. Fast decisions are important, but not if they are contradictory. In our experience, too many companies try to manage a cybersecurity incident in a decentralized fashion. An effective IR plan establishes clear roles and responsibilities across the organization, but we regularly see that organizations do not have a single leader for all incident response actions. It is a preventable mistake.
An “on-site commander” can coordinate disparate elements of the response. This may seem easy to anticipate and implement, but being able to understand and bring together functions as disparate as technical forensics and social media monitoring is not typically a skill set a single person would have. However, best practice has shown that having a single individual to manage the incident response ensures a much more thorough and consistent set of actions with clear lines of authority. In many organizations, this commander is drawn from the business continuity function, but he or she could easily be a senior IT security executive or a leader from the risk management function. CISOs, however, are often not best suited to the task as they need to concentrate on thinking strategically about the response while the incident commander coordinates the tactical actions.
Effective planning means predetermining how all business functions will work together: corporate communications, regulatory affairs, legal, compliance and audit, and business operations. This coordination, together with easily accessible IR documentation, ensures that the organization can react with greater agility during an incident.
Tighten Third-Party Collaboration
Effective IR plans should help improve an organization’s relationships with important third parties, such as law enforcement agencies and breach remediation and forensics experts.2 Often, a lack of planning or a reticence to air dirty laundry can significantly impede effective incident response. One financial services company had no contract with a third-party forensics firm. During a breach, it lost critical time that ran into days while procurement had to work through a new contract and the forensics vendor familiarized itself with the company’s network. These are all tasks that should have been done in advance.
In another example, the legal team at a North American company insisted that a felony had been committed due to a breach. The CISO, however, did not want to contact law enforcement officials for fear that they would delay business continuity operations in favor of preserving forensic evidence for trial. To make matters even worse, managers were not even sure which law enforcement agency to contact.
By contrast, when Hyundai Capital realized it had been breached, it immediately contacted the appropriate law enforcement agencies, who moved swiftly. The perpetrators were tracked down relatively quickly and the extent of the breach was limited.
Overcome the Shortfalls of Existing IR Plans
Many companies have an IR plan already. Indeed, some leading organizations invest serious time, money, and effort in these plans. In one typical organization, the IR plan ran to more than 80 pages and was professionally produced. Between them, team members took the equivalent of 20 months to draw it up.
Unfortunately, few plans are robust enough to truly deliver on the three objectives outlined earlier. They are often too generic when they should be guiding specific activities during a crisis. Key decision makers do not always have access to the plans, and the plan can rely too heavily on one or two go-to experts, who can become a point of failure if they are not available. Worst of all, we sometimes see locally optimized response plans, which can be useful for dealing with targeted attacks on specific business units, but are not effective for managing an incident across the whole business. Developing individual plans in silos also inhibits the sharing of relevant knowledge and best practices.
An effective IR plan addresses these shortfalls but, whether starting anew or building upon an existing effort, creating such a plan requires substantial work and dedicated project resources and should be treated as a formal initiative. Before drawing up the plan itself, best-practice organizations assess the existing response protocols and identify the most critical assets at risk. They also need to know which critical business functions will be called upon in an incident and assign the go-to person and the backup person (who should be just as well prepared) for each function. This group is then the core IR team.
Each primary function owner is responsible for maintaining his or her portion of the IR plan, with the IR team leader (who may be the CISO) ensuring that the plan and the cross-functional linkages are updated continuously. This setup cements the idea that the IR plan is not a special cybersecurity initiative but another business-as-usual practice.
Existing response protocols can be a useful framework from which to develop an IR plan template. Equipped with a solid understanding of the existing environment, organizations can assess how effective previous response efforts have been. For each incident, they should identify any problems that arose with the response, diagnose potential causes for failure, and create an exhaustive list of what can go wrong.
We discussed earlier in the book that organizations need to identify which information assets are most critical to business operations in order to be able to develop the data-specific actions to be taken. The IR team needs to talk to the key people from sales, marketing, operations, IT, security, regulatory affairs, and communications in order to understand the vulnerabilities and potential threats to these assets, the business impact if the asset is compromised, and the response required. This means that when the IR plan is needed, the response will be targeted and not generic. The war-gaming process that we discuss later is another important tool in understanding what is at risk.
Once the team has created the structure of the IR document, it should share the draft with the security team. This not only solicits valuable feedback from an eventual end user but also generates excitement for the tool.
The Components of a Robust IR Plan
There is no set template for an IR plan, though the most effective all contain an incident and asset taxonomy, define responsibilities and the war room setup, and—at their heart—have the playbook, broken down for every plausible scenario.
Incident and Asset Taxonomy
Even beyond the United States, organizations typically follow the incident categorization set by the U.S. National Institute of Standards and Technology. Broadly, this defines incidents as either unauthorized access, malicious code, denial of service, or inappropriate use. Adopting a common taxonomy enables institutions to share security intelligence with each other very easily, as well as standardize their own internal communications. Similarly, developing an internally consistent view on critical information assets will ensure the appropriate response based on the type of information compromised. For example, the response for a breach of personally identifiable customer information will differ from that for a breach of a merger and acquisition (M&A) strategy. Having a validated taxonomy of critical information assets is the starting point for reacting in the right way.
Responsibilities and War Room Setup
The war room is an essential IR tool. It is a well-worn concept for the military but applicable in a cyber-attack. A war room is a physical location with supporting infrastructure (IT connectivity, communications, security, and—importantly—snacks) where the preassigned IR team gathers to share knowledge, speed up decision making, and ensure unity of effort while responding to an incident.
Each member of the team, representing the critical business functions, has decision-making authority, but the on-site commander runs the war room, setting the reporting cadence and managing the decision-making process. It is also vital to have a scribe; someone to capture all the decisions that are made, the assumptions that lead to those decisions, the actions to be taken, and the external communications required. The scribe also logs all the outstanding issues to make sure nothing falls through the cracks. All this information is important evidence for stakeholders and regulators when the time comes to examine why the team made the decisions it did.
IR plans need to specify team structures in the event of an incident, individual roles and responsibilities (based on the assets at risk), escalation processes, and war-room protocols. For example, it is important to specify exactly when to involve executive leadership in the decision processes, when to activate a war room, and at what threshold executives should take decisive measures, such as isolating sections of the network or shutting down core applications. Operating models also document the all-important decision rights, for instance, who authorizes contacting law enforcement and which agencies to contact.
The Playbook
The response playbook itself consists of procedural guides including checklists for containment, eradication, and recovery, as well as guidelines for documenting the response in terms of governance, risk, and compliance. The detailed approaches will vary for each type of data and incident.
Even when incidents are of the same type (e.g., a malicious code breach), organizations will define their responses differently based on the types of information assets they believe are at risk. In other words, the type of data being compromised will determine response efforts because the business impact will be different. This is the most critical element of an IR plan and largely determines the success of the overall response.
For example, a company might have one set of response processes for the loss of confidential customer data and an entirely different set of processes for a loss of critical intellectual property (IP) even if both were caused by the same type of attack. The stakeholders are different in each case, and the resources a company will allocate to mitigation will vary.
For example, the goal when responding to a loss of customer data could be to identify the number of customers affected and the extent of data loss within four hours. Within eight hours the security team should have a good idea of who might be responsible for the theft and an estimate of the business impact. If, instead, the breach compromises sensitive M&A details, then the objective might be to determine the impact of each detail on the deal within 24 hours. This would then trigger a set of activities such as notifying key parties, including regulators, counterparty stakeholders, and investors.
The playbook should also clarify when notification should occur; this will vary by country, and even by state in the United States. By September 2014, 47 states had legislation that requires private and government entities to notify individuals if their personally identifiable information has been compromised. However, these laws differ in terms of who must comply, what “personally identifiable information” specifically means, what “compromised” means, and in terms of notice periods and exemptions. These state-level reporting requirements are in addition to any requirements from federal regulatory agencies.3
Even the team setup and operating model will vary by incident type, with roles and responsibilities assigned to specific individuals. Some sophisticated IR teams will assign individuals to deal with a particular jurisdiction or a specific regulatory body. The composition of such a team will therefore have to be flexible, depending on where the compromised records are located.
TEST THE PLAN USING WAR GAMES
An IR plan—even a well-designed one—that sits on the shelf is of limited use if the organization is not well versed in how to use it, and if it is not updated regularly. It is not a static document and should not be buried away in a file but rather woven into the fabric of the organization. The plan needs updating after every incident, and there should be continuous work to identify its weaknesses—that is, where things could go wrong—and to make the necessary adjustments.
Best-practice organizations make sure the plan is distributed widely to all relevant parts of the organization. It should be available in print, in digital formats, and on an internal web-based platform. But what really distinguishes a good IR plan from a great one is having the protocols embedded in the organization’s muscle memory through regular training and practice. War-gaming techniques are one powerful way of achieving this, though they are not a commitment to make lightly. A large company with multiple business units should seek to run such simulations quarterly, with a different scenario each time. A smaller company might run the exercise enterprise-wide twice a year. This may feel like a big time sink, but it is essential. The impact of responding badly or slowly to an (inevitable) attack is far too great.
The armed forces have long conducted war games to test their capabilities, reveal gaps in plans and build their leaders’ ability to make decisions in real time. Leading companies have embraced this concept and conduct cyber war games to help them ensure they have an acceptable answer when the CEO asks, “Are we ready?”
A cyber war game is different from traditional penetration testing, in which companies hire hackers to identify technical vulnerabilities such as unsecured network ports or externally facing programs that share too much information in the browser bar. A cyber war game is far more complex. It can yield insights into which information assets require protection, where there are vulnerabilities that attackers can exploit, and flaws in a corporation’s ability to respond to an attack, especially in those crucial areas of communications and decision-making processes.
As we mentioned earlier, the very act of setting up a war game starts a discussion between business and security managers about which types of information assets are most important, who would want to compromise them and what the implications of an attack could be in terms of loss of IP, loss of reputation, or business disruption. One public institution found out through designing a war game that while most of its IT security processes were oriented to preventing online fraud, its biggest risk was actually the loss of confidence associated with a public breach.
The analysis required to ensure that scenarios used in the game are realistic can highlight important security vulnerabilities. In preparing for a war game, a retail brokerage discovered that a large share of its most sensitive data was hosted on applications that had not undergone security reviews and that used out-of-date controls for authenticating users.
How to Run a War Game
The first step in producing a war game is to align on its scope and objectives. It should be organized around a business scenario rather than a purely technical breach and naturally the scenario should be relevant to the particular company in terms of both potential attack and the assets that are at risk. For example, a bank set up a scenario in which cyber-criminals used spear-phishing attacks to target high-net-worth customers in order to defraud them, while a high-tech company built a game based on a sophisticated attacker paying a corporate insider to install malware that enabled the theft of critical IP. Table 7.1 outlines the different decisions needed at different phases of the game.
TABLE 7.1 A War Game Tests Companies’ Processes and Nerve
| Phase | Description | Decisions Required |
| Initial indication of compromise | Employee reports that hackers have been bragging on underground websites that they have exfiltrated transaction data for five hedge fund clients with positions in the oil and gas industry Data from security systems and network logs suggests that there might have been a breach but data is inconclusive | Given what you know, do you contact affected customers or do you wait? Do you communicate anything to regulators? |
| Hacktivists issue ultimatum | Hacktivists initiate contact and claim they have ability to access weeks of trading data for bank’s largest clients As proof, they transmit one week’s transaction data for one client They promise to release all client data online unless bank publicly commits to cease all trading and “speculation” in the oil and gas sector | Do you ignore or react to the ultimatum? Do you contact the hacktivists and open a dialogue with them? Do you start cooperation with law enforcement? |
| Media inquiries | Respected news organization contacts bank and says they are in process of confirming reports that client data has been compromised via a breach in bank’s systems | What do you tell external media? Do you confirm reports? What do you communicate internally? Do you make any sort of broader communication to customers? |
| Initial forensics available | Security team confirms that bank is the source of the compromised data, and believes it has localized breach to 1 to 2 core trading systems CISO recommends shutting down affected systems to complete forensics—this will severely limit trading activity | Do you shut down affected systems? |
Companies need to decide how many scenarios to incorporate into the game, how sophisticated those scenarios will be, and how much participation will be required from each business function, especially from those who will design and run the war game. After choosing the scenarios, the team running the game identifies what response failures it needs to test for and creates the detailed script that the facilitators will use.
The game should simulate the experience of a real attack: participants receive incomplete information, and objectives may not be fully aligned (Figure 7.1). It should certainly involve participants from beyond information security and indeed beyond IT more broadly. It needs to include customer care, operations, marketing, legal, government affairs, and corporate communications. These sorts of war games happen in real time and can last anywhere from a day to a week or more, depending on the complexity of the scenarios. Throughout the course of the simulation, the facilitator will provide participants with information and what actions they will take. At each turn, the information the players receive depends on the actions they have just taken. Of course, no live systems are involved.
FIGURE 7.1 Base War-Game Scenario on High-Risk Events for the Business
The war game should reveal any flaws in the organization’s ability to respond to an attack. Specifically, it looks at three dimensions of the response: the speed of identifying and assessing the breach, the effectiveness of decision making in containing the breach, and the effectiveness of stakeholder communication.
Speed of Identifying and Assessing the Breach
One organization found that the processes used to address a breach were entirely dependent on e-mail and instant messaging. If the attack itself compromised those systems, then the company’s ability to respond would be severely hampered.
Effectiveness of Decision Making in Containing the Breach
A large multinational insurer discovered it had no guidelines for deciding when to shut down parts of its IT systems. During the game, senior executives ordered the technology team to sever external connectivity, even though this was not required and, in a real-world situation, would have prevented customers from accessing their accounts. In another example, business managers at a manufacturing company realized they had never thought through the implications if competitors or contractors got access to sensitive information. This meant they were not equipped to change negotiation strategies quickly if proprietary information about their cost structure was disclosed.
Effectiveness of Stakeholder Communication
A war game at the bank testing the spear-phishing scenario revealed that it had no guidelines for communicating with customers whose data had been compromised. In a real attack, the result would have been that those valuable high-net-worth customers would have received an impersonal generic e-mail rather than the tailored personalized communication they would expect.
With the war-game action over and the gaps apparent, the final and most important phase of the process converts the insights generated into actionable steps to improve the organization’s ability to respond to an attack. These typically include everything from implementing tools that increase visibility into attacks (such as an advanced analytics engine that detects anomalies that might be indicative of a zero-day attack), clarifying responsibilities, developing guidelines for making high-stakes decisions under pressure, and creating communications protocols that can be pulled off the shelf when required.
Most corporations can plan and conduct a game in 6 to 12 weeks, with an acceptable impact on managers. As we said, they are not a one-off exercise; the cybersecurity landscape is far too dynamic for any company to rest on its laurels. Although conducting this type of war game requires real effort and planning, it remains one of the most effective mechanisms for prioritizing which assets to protect, revealing vulnerabilities, identifying flaws in the ability to respond, and building the type of muscle memory required to make appropriate decisions in real time with limited information.
CONDUCT POSTMORTEMS ON REAL BREACHES TO IMPROVE IR PLAN
War games will not stop a breach occurring. Part of continuously upgrading and refining an IR plan is to incorporate the outcomes of responses to attacks when they do happen. This way, mistakes can be rectified and response protocols improved. Leading organizations already make sure that after a meaningful breach (i.e., one that triggers a response because data is compromised) they take a step back to examine how the IR plan worked. Few, however, conduct a full postmortem. Such a report should analyze the type of incident and determine whether it was categorized correctly (which of course determines the type of response), examine what exactly the attacker did, the specific steps taken by both the security team and other business functions, the degree and success of the response and the level of coordination, and the timeline of events in order to identify inefficiencies, errors, or areas where the IR plan lacked information.
Most companies allocate postmortem responsibility to the security operations team, but they tend to focus almost exclusively on the technical postmortem—that is, what sort of malware was used, which systems it tried to exploit etc. But this ignores how the organization responded, which is so important to stakeholders. Therefore the postmortem needs to be taken out of the hands of the CISO and given to someone with cross-functional responsibilities.
The postmortem team can also conduct what-if analysis to determine how different response actions at each phase might have affected the effectiveness of the attack. All this data then needs to be fed back and incorporated into the IR plan as part of an organization’s continuous testing and learning about how to respond to a cybersecurity incident.
● ● ●
A company can prioritize its information assets and determine the most effective set of defense mechanisms to put in place for each of its important information assets. It can integrate security into core business processes and change the behavior of frontline users. It can build security into its application and infrastructure architectures, and it can engage in active defense. These all make it less likely that a company will suffer a breach, but none can ensure that a company will not be breached. Therefore, achieving digital resilience requires the creation and aggressive testing of IR capabilities to respond to breaches, not only in technology but across all business functions.
Progress in this area requires that senior executives in almost every business area participate in war games to test and enhance response functions. In addition, the general counsel, the chief risk officer and the chief information officer should support the synchronization of the cybersecurity IR plan with other corporate crisis management plans.