Chapter 2. The Threat Intelligence Cycle
As established in Chapter 1, threat intelligence is not a data feed. Instead, threat intelligence is a system. Good threat intelligence teams have a process in place that gives them the ability to continuously adjust to new threats and quickly incorporate new data sources into their intelligence process. Almost all threat intelligence organizations use the intelligence cycle model, with some variation in the terms and numbers of phases.
The Intelligence Cycle
The most commonly used threat intelligence model is the intelligence cycle, shown in Figure 2-1, or a variant on this model.
Figure 2-1. The Intelligence Cycle
This is the model that is used by military intelligence, and it consists of five parts, some of which have already been discussed:
- Planning and Direction
- Collection
- Processing
- Production
- Dissemination
At the core of the intelligence cycle is the mission. The five components of the intelligence cycle revolve around helping the organization succeed in its mission. Note that no one part of the intelligence cycle is more important than the other parts. In order for a threat intelligence program to be effective, all components of the threat intelligence cycle have to work equally well.
Intelligence Requirements
The flow of the intelligence cycle allows the threat intelligence team to sift through the incredible amounts of data that is collected by the organization and produce actionable intelligence that makes the security team more effective and improves the security of the organization. The process is truly circular in nature. So, while it may seem obvious to start the intelligence cycle by talking about requirements, as discussed in Chapter 1, requirements generally stem from collected data and analysis of that data. A requirement is a subject about which the threat intelligence team has to collect information or produce reporting. Requirements are often, though not always, requested by someone outside of the threat intelligence organization.
Because the process has to start somewhere, let’s start with the requirements phase. Producing good requirements involves asking good questions and properly prioritizing the requirements that are determined based on the responses to those questions. Military intelligence uses the term Priority Intelligence Requirements (PIR) to refer to those requirements that are most critical to the organization, or most time sensitive.
Whether being used with military intelligence nomenclature or developing a different method, every organization has to determine how it is going to prioritize intelligence requirements. The simple fact is that no matter how many people threat intelligence team has or how large the budget is, there are simply more intelligence requirements than there is time to resolve them all. Therefore, it is important to ensure that the most critical intelligence requirements are resolved first.
High priority intelligence requirements should primarily be those that are most closely tied to the core mission of the organization, however that is not always the case. There may be time-sensitive requirements that need to receive a higher priority simply because there is a smaller window in which to get answers. For example, an organization that is sponsoring FIFA’s World Cup may be concerned about both physical and cyber threats surrounding the event, as any such incidents could damage the reputation of their brand. Those intelligence requirements would receive a higher priority because of the fixed time period for their relevance as well as the high profile of the World Cup. Lower priority intelligence requirements tend to be more technical, and ongoing. An example of a lower priority intelligence requirement might include an organization monitoring underground forums for mentions of network blocks assigned to that organization. This is an ongoing and repetitive task that is not time sensitive.
In addition to being prioritized, intelligence requirements must be specific in order to be effective. Crafting effective intelligence requirements is almost an art form and is one of the reasons that good threat intelligence analysts are in such high demand. According the Army’s Intelligence Officer’s Handbook a good intelligence requirement has three components:1
- It asks a single question
- Focuses on a specific fact, event, or activity
- Provides intelligence required to support a single decision.
In other words, broad questions such as “What are all the threats against our organization and what tools are they using?” or “Who is talking about our organization on the underground forums, what are they saying, and are they a threat?” are not effective intelligence requirements. Instead, those questions can be broken down and refined into more close-ended requirements that can be readily satisfied. Examples of better intelligence requirements based on the broader questions are “What attackers are currently targeting organizations in our sector?” “What are the current TTPs associated with the Syrian Electronic Army?”
The idea of focused requirements may seem counterintuitive, especially considering that most threat intelligence teams are going to be understaffed and underfunded. Focused intelligence requirements, even if there are significantly more of them, actually enable a threat intelligence organization to be more effective. Narrower requirements allow the analysts to get a specific answer without having to guess what the original intent was behind the question. So, while there are more requirements to respond to, the threat intelligence team is able to respond to them faster and in a more complete fashion.
Collection
Collection is the phase of the threat intelligence cycle with which security teams and threat analysts are probably the most familiar. Collection is an inherent part of almost any security program—gathering log data from as many sources within an organization is seen as a critical component to success. Collection is the process of gathering data to fulfill the requirements.
Most organizations rely heavily on their Security Incident and Event Manager (SIEM) to act as the collection point for both security and threat intelligence purposes. There is nothing wrong with that, and SIEMs are very powerful tools. However, there are also limits to SIEM collection—starting with the fact that it is log centric, which can lead to limited thinking when it comes to collection. After all, if an organization’s primary collection source is seen primarily as a log aggregator, the collection phase is going to focus on logs.
True threat intelligence collection requires a breadth of sources. Log data is an important source, but it should only be one type of data that is included in the data collection process. It is important for a threat intelligence organization to think about other ways to collect data that may not fit into a traditional log format. For example, there is very valuable intelligence to be gained by mining NetFlow data, but NetFlow data doesn’t always fit nicely into a SIEM architecture. Collecting DNS resolution data for passive DNS analysis can also be very valuable, but again doesn’t always fit into a SIEM-based structure.
The good news is that there are a growing number of tools that allow for collection of unusual data sources in their raw form and still allow those sources to be correlated against log data to look for potential threats.
The scope of the data to be collected is dependent on the requirements that are laid out in the planning and requirements phase. That may seem obvious, but it is not always as easy as it sounds. One example of a problem that often arises is that organizations are concerned about the security, both physical and cyber, of remote offices. A reasonable set of requirements to answer this priority might be, “Based on recent examples, what are some of the physical security threats to our offices in the Philippines?” or “What attack groups are active the Philippines at this time?”
To effectively respond to those requirements may involve collecting news stories from local or international papers. The collection point needs to be able to ingest that information along with basic information like the addresses of any offices in the Philippines. Similarly, an organization’s threat intelligence provider may share a list of attack groups currently active in the Philippines along with their associated TTPs, but that list probably won’t be in log format.
A threat intelligence team should constantly be expanding the idea of what a threat to the organization looks like. It should be trying to find new sources of data to ingest, which requires a platform that can easily adapt as these sources continue to grow.
That breadth of intelligence shouldn’t apply just to intelligence generated from internal sources. Any third-party threat intelligence providers that an organization uses should have that same goal. Many threat intelligence providers are stuck in the mind-set that indicators are exclusively:
- IP addresses
- Domains
- File hashes
Again, these are useful indicators and will be important for any organization that is just starting the process of building a threat intelligence program. However, there is much more data that a third-party threat intelligence provider can deliver. Other indicators that a third-party provider can deliver include vulnerability information, account numbers associated with their customers (either reward cards or company credit cards), email addresses and subject lines associated with spam or phishing campaigns, proprietary code leaked to the internet, and more. It is important to partner with a third-party provider that can grow with an organization and help an organization expand its capabilities.
Processing
Even though they are distinct components of the threat intelligence cycle, analysis and processing often get lumped together because, in most organizations, these two tasks are carried out in the same platform. Whether an organization is using a SIEM for processing log and network data or aggregating threat intelligence into a Threat Intelligence Platform (TIP), for the most part the threat analyst team is relying on an underlying platform to handle much of the initial processing of collected data.
There is nothing wrong with that approach—in fact, in most cases, relying on anything but a platform that automates processing and correlation doesn’t make sense. There are simply too many data sources for a threat intelligence team to be able to analyze manually. An organization of any size is going to have millions of log events and hundreds of thousands of indicators to process each day. It is important to rely on correlation and automation to take the first pass at identifying potential threats.
SIEMs and TIPs
As mentioned earlier, today’s threat intelligence teams primarily rely on two platforms for analysis and production. The first is the SIEM, typically those are platforms like ArcSight, LogRhythm, QRadar, or Splunk. The second platform, not as widely used today but gaining market share is the TIP. Some of the most commonly deployed TIPs are Anomali, ThreatConnect, and ThreatQ. Both SIEMs and TIPs have their pluses and minuses, but they both can serve a vital role in the threat intelligence life cycle.
Security teams beginning the process of building out a threat intelligence program generally start with a SIEM. SIEMs are very powerful because they take a morass of data and provide a structure that allows for easy correlation. By dumping log data and data feeds into the same platform with all the data conforming to a singular framework, threat intelligence analysts can easily write correlation rules. In fact, most SIEM platforms offer a large number of out-of-the-box correlation rules that can be used for security, compliance, auditing, and other purposes. So, while a SIEM does require a great deal of “care and feeding” to operate in an efficient manner, a threat intelligence team can get up and running on a SIEM platform in a relatively quick time frame.
However, the SIEM’s greatest strength can also be a drawback: the rigid framework of the data that is populated into the SIEM can limit the data types that can be used in the automated analysis. Again, for organizations just starting to build out a threat intelligence program, that is probably OK. But as the threat intelligence program matures, the organization may need to move beyond the SIEM.
TIPs help to fill the gap that is left by some SIEMs, at least when it comes to threat intelligence ingestion. Because TIPs aggregate multiple types of sources of threat intelligence, they tend to be more flexible in their data structures than SIEMs. This allows them to ingest threat intelligence in different forms, often structured and unstructured. Despite the less structured nature of the data that is inputted into the platform, TIPs can produce results in a structured manner that can then be delivered to SIEMs or other platforms to provide context and actionable results.
The strength of the TIP platform is its flexibility. However, TIPs are not designed to ingest large amounts of log data—their focus is strictly on threat intelligence. This means that threat analysts are still required to manage at least two platforms, the SIEM and the TIP.
A Better Solution
A better solution, for more advanced threat intelligence programs, is to bypass the idea of structured data entirely and instead to drop all log and threat intelligence data into a single unstructured platform. Using tools like the Elastic Stack, which allow organizations to include all sorts of data sets, in any format, and analyze those data sets, no matter how disparate they seem, can enable threat intelligence analysts to make connections that are not able to be made by a SIEM.
In order to challenge systems like Elastic Stack, a few SIEM vendors have adopted this type of backend infrastructure, creating a greater flexibility for their customers. Surprisingly, this approach can also allow for greater scalability than the traditional, more structured, SIEM solutions. That being said, working with unstructured data usually involves more work by the threat intelligence team or an organization’s Security Operations Center (SOC), as often queries will need to be built from scratch and correlation rules will need to be developed by the threat analyst team. An Elastic Stack solution, or other similar solutions, can take significantly longer to get up and running than a traditional SIEM and may require more adjustments over time.
That is not to say that this type of flexibility is not worth pursuing. Some of the largest threat intelligence programs in the world use this type of solution for threat analysis. It is simply important to be aware of the challenges and time commitment that is often required to make these types of solutions work.
Analysis Strategy
From a threat intelligence program perspective there are two goals to be achieved from the analysis of collected data. The first goal is to uncover potential security issues and alert the relevant teams about them. The second goal is to create responses to the ongoing intelligence requirements.
Handling immediate security alerts generated by analysis of collected data is generally handled by the Security Operations Center (SOC) analysts (though, in many organizations there is no difference between SOC analysts and threat intelligence analysts). This type of reporting involves correlating third-party threat intelligence against logs generated by security systems looking for matches against common Indicator of Compromise (IOC) types, such as IP addresses, domain names, or file hashes.
This type of reporting is a way to show immediate value from third-party threat intelligence providers. Most SOCs are overwhelmed by the number of alerts being generated by the various security platforms in the organization. Assuming a third-party threat intelligence provider has done a good job of filtering the indictors they are sending to their customers, and they can provide context around those indicators, third-party threat intelligence can help SOC analysts better prioritize alerts and respond to immediate threats more efficiently.
Even the best SOC analysts can only respond to about eight security incidents per day.2 A SOC of any size will get hundreds of incidents per day, which means that a lot of security incidents will be ignored. Correlating contextualized threat intelligence from a third-party provider against internal logs helps the SOC identify and prioritize critical incidents based on the criticality of the indicators provided by the provider.
Automated Correlation
The importance of automated correlation cannot be overstated in a modern SOC, and it is a critical part of a threat intelligence program. Tying together different aspects of an attack by linking different sources of data through timestamps or common indicators allows threat intelligence teams to extract intelligence in a much faster manner than before these tools were available.
But automated correlation should not be the only focus of the threat intelligence program. This type of response is almost entirely reactive. The other side of the threat intelligence program, creating and responding to intelligence requirements, allows the threat intelligence program to be more proactive.
Clearly defined intelligence priorities, help build a security strategy and can help build both short-term and long-term goals. Depending on how it is structured, the same collection system can be used by both the SOC team (short-term) and threat intelligence analysts (long-term).
For example, a SOC analyst may notice unusual traffic to an external IP address. The traffic doesn’t seem malicious on the surface, but it is odd enough that it is worth creating an intelligence requirement for deeper analysis. The threat intelligence team can review collected data to see if that pattern has been repeated elsewhere, or at an earlier time, in the network. The threat intelligence team can also determine if there have been indicators of malicious behavior associated with the unusual traffic pattern. Beyond what is available in the collected systems, threat analysts can reach out to other organizations to see if anyone else has experienced that pattern and if those organizations had uncovered any malicious activity associated with it.
Again, this is where actionable and contextual intelligence from third-party providers is important. If that unusual traffic pattern is associated with malicious activity, a threat intelligence provider should codify it in such a way that the two activities are tied together. That way, when the next SOC analyst notices the pattern, all of the relevant information will be automatically connected, and the analyst can take immediate action.
Production
Up to this point in the intelligence cycle the discussion has revolved around raw data. Even third-party threat intelligence ingested into platforms is just data at this point. The production phase of the threat intelligence is where the raw data becomes threat intelligence. Production is the process of turning the raw collected data into threat intelligence.
Production can take many forms, each with a specific purpose and audience in mind. Traditionally, the production phase involved the creation of reports that could be delivered to customers as part of the dissemination and feedback phase. Report production is still an important part of the threat intelligence cycle and a critical function of the threat intelligence team.
Reports provide a peer-reviewed, in-depth look at a threat to the organization that is generally in response to an intelligence requirement. But production is not limited to just reporting. Processing the raw collected data can result in the production of internal threat lists that can be distributed to other teams within the organization to create firewall rules, domains to block on a proxy, or signatures that can be incorporated into incident response platforms.
Bias and Groupthink
The review process is an important part of intelligence production. All humans are subject to biases—the best threat intelligence analysts try to be aware of their biases and to keep those biases from creeping into their reporting. However, even the best analyst can sometimes unwittingly act based on biases.
Having another analyst review reporting before it is released can help to alleviate the risks associated with biases in reporting. Ideally, the reviewer should be someone in a separate group, to avoid any biases associated with groupthink. For large threat intelligence teams broken into smaller groups, having an analyst from one group review reporting from another group should be sufficient. Smaller threat intelligence teams may need to designate reviewers in another group, such as the SOC, in order to ensure that biases and groupthink are kept to a minimum in threat intelligence reporting.
Production is not just creating reports. It is about getting the data in a format that can be easily consumed by the customers of the threat intelligence team in a timely manner and tracking the delivery of these finished pieces of intelligence.
Ticketing Systems
In order for the different stages of the intelligence cycle to function as a whole there must be something that connects them all. There has to be a way to get from the requirements phase through processing, production, and dissemination in a cohesive manner.
The most common way to do this is through a ticketing system, like JIRA, Service Desk, Remedy, or ServiceNow. Ticketing systems generally provide the flexibility for organizations to use them to create and respond to intelligence requirements while also allowing the SOC to use them to report the results of an investigation or put in an IT management request.
Beyond their basic purposes, ticketing systems can also be used by the threat intelligence team as a way to track more in-depth reporting, especially in response to PIRs. Because ticketing systems connect to both SIEMs and TIPs, as well as other security systems, supporting data from those platforms can be passed directly to the ticketing system. Which means that the response to the PIR can contain the write-up as well as any necessary raw data.
Ticketing systems have a few other advantages that make them attractive in the production process. The first is that they can be used as part of the review process. Generally, when a response to a PIR is written up, it should be reviewed by at least one other analyst before dissemination, and often there is a multi-step review process. All of this can be tracked within the ticketing system.
Secondly, a ticketing system is great for tracking time-sensitive information. When requirements are created, a deadline for response should be included in the ticketing system. The system will then automatically alert the relevant parties as the deadline approaches and even produce dashboards that allow the manager of the threat intelligence team to track the progress of the intelligence requirements.
Finally, going back to the requirements creation process, ticketing systems allow the threat intelligence team to create custom priority levels for tickets. Prioritization is an important part of the intelligence requirements process. Using a ticket system to create custom prioritization levels and then tracking the process of those requirements will ensure that the threat intelligence team is devoting resources to the right priorities and doing so in a timely fashion.
Dissemination
Dissemination is the process of distributing finished threat intelligence to the original customer as well as anyone else in the organization who requires access. Whether it is a finished report or a threat list, it is important to ensure that it is delivered to the original requestor in a timely fashion and any feedback from the customer is processed. This is usually done through a ticketing system, as described above.
Keep in mind that intelligence reporting doesn’t necessarily have to be written up inside the ticketing system—the ticketing system can just be used for tracking purposes. In fact, in many cases, it makes more sense to use standard office tools, such as Microsoft Word, to deliver reporting based on requirements.
The reason it makes sense to use widely available office tools is that, if a threat intelligence team is successful, many of its requirements will originate from outside the team. Most organizations are not going to require that executives or members of the board of directors use a ticketing system to generate intelligence requirements or read the resulting reporting. Forcing customers outside of the threat intelligence or SOC teams to use a ticketing system to create intelligence requirements may result in fewer requirements being created and create the perception that the threat intelligence team is less valuable.
Customers versus Coworkers
This book uses the term customers to refer to other people in the organization that submit requirements to the threat intelligence team.
Threat intelligence organizations often adopt the nomenclature of referring to the leadership that creates the intelligence requirements as customers. In this case, a customer is not someone external to the larger organization, instead it is a frame of reference for the threat analysts.
The term customer generally implies that there is an obligation to provide a certain level of service, and can serve as a reminder of that expected service.
Threat intelligence teams often set up an email address that people can use to submit intelligence requests. That email address can automatically open a ticket and be used to track intelligence requests and PIRs, even if the ticketing system is invisible to the customer.
Aside from tracking, using a ticketing system has another advantage—it helps to ensure that the responses to these requests are fed back into the data collection process. The responses, when appropriate, should be stored with other collected data and used to help respond to future intelligence requests, in other words, they become part of the intelligence cycle.
When a response to an intelligence requirement has been sent to the original requestor, it is important to reach out to requestor to ensure that the request was satisfied and there are no follow-up requirements. This is the part of the intelligence cycle that is most often ignored by threat intelligence teams. Organizations are overworked and already have too many intelligence requirements to answer in a timely fashion. It is easier to just assume that if there are any questions or follow-up the customer will reach out. But that is not always the case—the customer may be confused by a response but not want to reach out, or the customer just gets busy and doesn’t have time to review.
Reaching out to ensure that the requirement was properly responded to often generates additional requirements and can help improve processes within the threat intelligence team. Especially when a customer is confused about language or terms used in the response. Responding to that feedback can help the threat intelligence team become more precise or get better at explaining complex technical terms.
Tracking intelligence requirements through the entire cycle, from creation to dissemination and feedback, is important. It is also important to create dashboards documenting the overall success of the threat intelligence team. The idea of creating dashboards for intelligence requirements is anathema to many threat intelligence professionals, but it is a quick and easily distillable way to illustrate the value of the threat intelligence team. Like it or not, security and threat intelligence teams cannot be seen as separate from the business, instead they must be seen as a core of the business and must be able to demonstrate value in terms that those on the business side of the house can understand. Dashboards are one way to do that.
Of course, the other way for the threat intelligence team to show value is to follow up with their customers during the dissemination and feedback phase of the intelligence cycle. Ensuring that reports are read and understood, getting feedback, and channeling that feedback back into the intelligence cycle also demonstrate the value of the threat intelligence program.
Summary
There are a number of of different threat intelligence models that an organization can follow, but most use the threat intelligence cycle. This cycle involves five phases that all build on each other in a continuous loop: Planning and Direction, Collection, Processing, Production, and Dissemination. All these should revolve around the overall mission of the organization.
Gathering requirements and prioritizing those requirements is an important task for the threat intelligence team. Most people will not know how to properly create intelligence requirements. It is up the threat intelligence team to get to the heart of the questions being asked and create useful intelligence requirements based on those questions.
Collection should be from as many sources, internal and external, as possible and in as many forms as possible. Organizations just starting a threat intelligence program often rely on SIEMs as the collection point, but it is important to make sure that the SIEM will meet all the collection needs of the organization. At least initially, the collection platform may also serve as the analysis and production platform.
The threat intelligence team doesn’t just take in requirements; it also has to produce responses during the dissemination phase. Tracking intelligence requirements through the entire intelligence cycle usually requires a ticketing system of some sort. The ticketing system can track the creation, analysis, and production of the intelligence requirement, but the dissemination is often done using standard office systems, such as Microsoft Word. Even when documents are delivered in a standard format they should be tracked, and it is important to follow up with customers to see if there is any feedback or new intelligence requirements that originated from the reporting.
1 U.S. Army, Intelligence Officer’s Handbook, FM 34-8-2 (Washington, D.C.: Government Printing Office, 1998), Appendix D.
2 Goldfarb, Joshua, “Spear Alerting: Improving Efficiency of Security Operations and Incident Response”, SecurityWeek, 15 Dec 2014, accessed 31 Jan 2017.
3 Weedon, Jen, William Nuland, and Alex Stamos, “Information Operations and Facebook”, Facebook, 27 Apr 2017, accessed 19 June 2017.