Chapter 8: Microsoft Sentinel

Security Information and Event Management (SIEM) combines two solutions that were previously separate, Security Information Management (SIM) and Security Event Management (SEM).

We have already mentioned that large organizations rely on SIEM solutions. And Microsoft's SIEM solution for the cloud is Microsoft Sentinel. But let's first take a step back and discuss what SIEM is and what functionalities it should have.

We will be covering the following topics in this chapter:

• Introduction to SIEM
• Getting started with Microsoft Sentinel
• Creating workbooks
• Using threat hunting and notebooks

Introduction to SIEM

Many security compliance standards require long-term storage, where security-related logs should be kept for long periods of time. This varies from one compliance standard to another and can be up to many years. This is where SIM comes into the picture: long-term storage where all security-related logs are stored for analysis and reports.

When we speak of SEM, we tend to be talking about live data streaming rather than long-term event tracking. SEM's focus is on real-time monitoring; it aims to correlate events using notifications and dashboards. When we combine these two, we have SIEM, which tries to live stream all security-related logs and keep them for the long term. With this approach, we have a real-time monitoring and reporting tool in one solution.

When discussing the functionalities required, we have a few checkboxes that SIEM must tick:

• Data aggregation: Logs across different systems are kept in a single place. This can include network, application, server, and database logs, to name a few.
• Dashboards: All aggregated data can be used to create charts. These charts can help us to visually detect patterns or anomalies.
• Alerts: Aggregated data is analyzed automatically, and any detected anomaly becomes an alert. Alerts are then sent to individuals or teams that must be aware of them, or act on them.
• Correlation: One of SIEM's responsibilities is to provide a meaningful connection between common attributes and events. This is also where data aggregation comes in because it helps to identify connected events across different log types. A single line in a database log may not mean much, but if it's combined with logs from the network and the application, it can help us prevent a disaster.
• Retention: As mentioned, one of the compliance requirements is to keep data for extended periods of time. But this also helps us to establish patterns over long periods and detect anomalies more easily.
• Forensic analysis: Once we are aware of a security issue, SIEM is used to analyze events and detect how and why the issue happened. This helps us to neutralize damage and prevent the issue from repeating.

To summarize, SIEM should have the ability to receive different data types in real time, provide meaning to received data, and store it for an extended period of time. Received data is then analyzed to find patterns, detect anomalies, and help us prevent or stop security incidents.

So, let's see how Microsoft Sentinel addresses these requirements.

Getting started with Microsoft Sentinel

Microsoft Sentinel is Microsoft's SIEM solution in the cloud. As cloud computing continues to revolutionize how we do IT, SIEM must evolve to address the new challenges that the changes to IT create. Microsoft Sentinel is a scalable cloud solution that offers intelligent security and threat analytics. On top of that, Microsoft Sentinel provides threat visibility and alerting, along with proactive threat hunting and responses.

So, if we look carefully, all the checkboxes for SIEM are ticked.

The pricing model for Microsoft Sentinel comes with two options:

• Pay-As-You-Go: In Pay-As-You-Go, billing is done per GB of data ingested to Microsoft Sentinel.

  Important Note

  At the time of writing, the price in the Pay-As-You-Go model was $2.60 per ingested GB.

• Capacity reservation: Capacity reservation offers different tiers with varying amounts of data reserved. Reservation creates a commitment and billing is done per tier, even if we don't use the reserved capacity. However, reservation provides a discount on ingested data and is a good option for organizations that expect large amounts of data.

The following diagram shows the capacity reservation prices for Microsoft Sentinel at the time of writing:

Figure 8.1 – Microsoft Sentinel pricing

If ingested data exceeds the reservation limit, further billing is done based on the Pay-As-You-Go model. For example, if we have a reserved capacity of 100 GB and we ingest 112 GB, we will pay the tier price for the reserved capacity up to 100 GB and will also pay for the additional 12 GB for the data that exceeds the reservation.

It's interesting to reflect on pricing presented in the first edition of this book and how it changed in 2 years. The price decreased in the region of 5%, but I find capacity options more interesting. 2 years ago, the maximum capacity was 500 GB per day, while today we have a capacity of up to 5,000 GB per day. This perfectly paints the picture of constant data growth and how data is becoming bigger and bigger right before our eyes.

When enabling Microsoft Sentinel, we need to define the Log Analytics workspace that will be used for storing data. We can either create a new Log Analytics workspace or use an existing one.

Microsoft Sentinel uses Log Analytics for storing data. The price for Microsoft Sentinel does not include charges for Log Analytics.

Important Note

An additional charge for Log Analytics will be incurred for ingested data. Information about pricing can be found at https://azure.microsoft.com/en-us/pricing/details/monitor/.

In the following screenshot, we can see all the pricing options for Microsoft Sentinel at the time of writing:

Figure 8.2 – Changing the Microsoft Sentinel pricing tier

Data retention is configured in Log Analytics Workspace. Log Analytics Workspace default retention is 30 days, but it can be increased to a period of up to 2 years. However, keep in mind that changing the retention period will create additional costs as this can increase the amount of data significantly.

Important Note

Microsoft Sentinel customers can increase data retention to 90 days for free. This only applies to a Log Analytics workspace connected to Microsoft Sentinel.

Now, let's see how Microsoft Sentinel does everything that SIEM should be able to do. We will analyze all the requirements one by one and see how they are satisfied by Microsoft Sentinel.

Let's start with data connectors and retention.

Configuring data connectors and retention

One of SIEM's requirements is data aggregation. However, data aggregation doesn't just mean collecting data, but also the ability to collect data from multiple sources. Microsoft Sentinel does this really well and has many integrated connectors. We can see how much attention Microsoft is paying to this part when comparing the number of available data connectors between the two editions of this book. In the first edition, it was mentioned that 32 data connectors were available. Today, we have 120 of them, an increase of over 350% in 2 years. A lot of the connectors are for different Microsoft sources, such as Azure Active Directory, Azure Active Directory Identity Protection, Microsoft Defender for Cloud, Microsoft Cloud App Security, or Office 365, to name but a few. However, there is also a number of connectors for data sources outside the Microsoft ecosystem, such as Amazon Web Services, Barracuda Firewalls, Cisco, Citrix, and Palo Alto Networks.

Important Note

For more information on connectors, you can refer to the following link: https://docs.microsoft.com/en-us/azure/sentinel/connect-data-sources.

The data connectors page is shown in the following screenshot:

Figure 8.3 – Microsoft Sentinel data connectors

All data connectors include step-by-step instructions explaining how to configure data sources to send data. It's important to mention that all the data is stored in Log Analytics. Instructions vary from data source to data source. Most Microsoft data sources can be added by just enabling a connection from one service to another. Other data sources require the installation of an agent or editing of the endpoint's configuration.

There are many default connectors in Microsoft Sentinel. Besides the obvious Microsoft connectors with services in Azure and Office 365, we have many connectors for on-premises services as well. But it doesn't stop there, and many other connectors are available, including Amazon Web Services, Barracuda, Cisco, Palo Alto, F5, and Symantec.

Once the data is imported into Log Analytics and is ready to be used in Microsoft Sentinel, that is when the real work starts.

Working with Microsoft Sentinel dashboards

After the data is gathered, the next step is to display data using various dashboards. Dashboards visually present data using Key Performance Indicators (KPIs), metrics, and key data points in order to monitor security events. Presenting data visually helps set up baseline patterns and detect anomalies.

The following screenshot shows events and alerts over time:

Figure 8.4 – Events and alerts dashboard

In this screenshot, we can see how the baseline is established. There is a similar number of events over time. Any sudden increase or decrease would be an anomaly that we would need to investigate.

The Events and alerts over time dashboard uses metrics to display data, but we can also use KPIs to create different types of dashboards. The following diagram shows anomalies in the data source:

Figure 8.5 – Anomalies dashboard

These two examples represent default dashboards that are available when Microsoft Sentinel is enabled. We can also create custom dashboards, based on the requirements and the KPIs defined.

However, this is only the first step in detecting something that's out of the ordinary. We need additional steps to automate the process.

Setting up rules and alerts

The only problem with dashboards is that they are useful if someone is watching them. Data is displayed visually, and we can detect issues only if we are monitoring dashboards at all times. But what happens when no one is watching? For these situations, we define rules and alerts.

Using rules, we can define a baseline and send notifications (or even automate responses) when any type of anomaly appears. In Microsoft Sentinel, we can create custom rules on the Analytics blade, with three types of rules available:

• The first type of rule is the Microsoft incident creation rule, where we can select from a list of predefined analytic rules. The rules here are Microsoft Cloud App Security, Microsoft Defender for Cloud, Azure Advanced Threat Protection, Azure Active Directory Identity Protection, and Microsoft Defender Advanced Threat Protection. The only other option is to select the severity of the incident that will be tracked.
• The second type of rule is Scheduled query rule. We have more options here, and we can define rules that track basically anything. The only limitation we have here is our data. The more data we have, the more things we can track. Using Kusto Query Language, we can create custom rules and check for any type of information as long as the data is already in the Log Analytics workspace.
• The third type of rule is NRT query rule. NRT stands for Near Real Time and the main difference between NRT and Scheduled is in the time delay. Where Scheduled rules have a time delay of up to 5 minutes, the NRT delay is 2 minutes. This is the type of rule we want to use when time is of the essence, and we want to detect an anomaly as soon as possible.

To create a custom query rule, the following steps are required:

1. We need to define a name, select the tactics, and set the severity we want to detect. There are several tactics options we can choose from: Initial Access, Execution, Persistence, Privileged Escalation, Defense Evasion, Credential Access, Discovery, Lateral Movement, Collection, Exfiltration, Command and Control, and Impact.

Optionally, we can add a description. Adding a description is recommended because it can help us track down rules and detect their purpose. An example is shown in the following screenshot:

Figure 8.6 – Creating a new analytics rule

2. Next, we need to set the rule's logic. In this part, we need to set up the rule query using Kusto Query Language. The query will be executed against data in Log Analytics in order to detect any events that represent a threat or an issue. The query needs to be correct because the syntax will be checked. A failed syntax check will result in the query failing to proceed. An example of a query in a custom rule is shown in the following screenshot:

Figure 8.7 – Defining an analytic rule query

The query used in this example is tracking resources on the account that were logged in the last 24 hours, which you can see here:

let GetAllHostsbyAccount = (v_Account_Name:string){

  SigninLogs

  | extend v_Account_Name = case(

  v_Account_Name has '@', tostring(split(v_Account_Name, '@')[0]),

  v_Account_Name has '', tostring(split(v_Account_Name, '')[1]),

  v_Account_Name

  )

  | where UserPrincipalName contains v_Account_Name

  | extend RemoteHost = tolower(tostring(parsejson(DeviceDetail.['displayName'])))

  | extend OS = DeviceDetail.operatingSystem, Browser = DeviceDetail.browser

  | extend StatusCode = tostring(Status.errorCode), StatusDetails = tostring(Status.additionalDetails)

  | extend State = tostring(LocationDetails.state), City = tostring(LocationDetails.city)

  | extend info = pack('UserDisplayName', UserDisplayName, 'UserPrincipalName', UserPrincipalName, 'AppDisplayName', AppDisplayName, 'ClientAppUsed', ClientAppUsed, 'Browser', tostring(Browser), 'IPAddress', IPAddress, 'ResultType', ResultType, 'ResultDescription', ResultDescription, 'Location', Location, 'State', State, 'City', City, 'StatusCode', StatusCode, 'StatusDetails', StatusDetails)

  | summarize min(TimeGenerated), max(TimeGenerated), Host_Aux_info = makeset(info) by RemoteHost, tostring(OS)

  | project min_TimeGenerated, max_TimeGenerated, RemoteHost, OS, Host_Aux_info

  | top 10 by min_TimeGenerated desc nulls last

  | project-rename Host_UnstructuredName=RemoteHost, Host_OSVersion=OS

  };

  // change <Name> value below

  GetAllHostsbyAccount('<Name>')

3. We can test this query either by selecting View Query Results (which will take us to Log Analytics Workspace) or by selecting Test with current data in the Results simulation part of the blade, as shown in the following screenshot:

Figure 8.8 – Testing query results

4. Once the query is defined, we need to create the schedule. The schedule defines how often the query is executed and the data that it's executed against. We also define the threshold for when an event becomes an alert. For example, a failed login attempt is just an event. But if this event repeats over time, then it becomes an alert.

The following screenshot shows an example of a schedule and a threshold:

Figure 8.9 – Analytic rule scheduling

Several optional settings allow us to enhance alerts with entity mapping, custom details, or alert details. We can also select event grouping and choose whether we want a query to stop running if an alert is generated.

5. Next, we move to Incident settings (in preview at the time of writing) where we can choose to create incident form alerts and set alert grouping, as shown in the following screenshot:

Figure 8.10 – Analytic rule incident settings

6. In the last step, we can define what will happen when the alert is triggered. Similar to workflow automation in Microsoft Defender for Cloud (in Chapter 7, Microsoft Defender for Cloud), Logic apps are used to create automated responses. These responses can either be notifications (to users or groups of users) or automated responses that will react to stop or prevent the threat.

Figure 8.11 – Automated response configuration

To have an automated response, we need to create playbooks. Let's take a look at how playbooks are created in Microsoft Sentinel.

Microsoft Sentinel automation

Under Automation in Microsoft Sentinel, we create playbooks that can provide an automated response if an alert is triggered. Responses can vary from blocking and preventing threats to sending notifications, depending on the severity.

We can either create a playbook from scratch or select an existing template. Let's take a look at the templates and select Block AAD user – Alert, as in the following example:

Figure 8.12 – Playbook templates

If we select a template and choose to create a playbook, it will take us to a new blade. We need to select a subscription, resource group, region, and playbook name. An example is shown in the following screenshot:

Figure 8.13 – Automated response

Optionally, we can select to send diagnostics logs to Log Analytics and configure different integrations.

Next, we can see connectors that will be created for the playbook in question, as shown:

Figure 8.14 – Playbook connectors

Finally, we review the settings and create a playbook.

Figure 8.15 – Reviewing and creating a playbook

Note that the connection created needs to be authorized once deployment is done. And now we have a way to automatically respond to threats and incidents.

But in modern cybersecurity, this may not be enough. Let's look at other options in Microsoft Sentinel.

Creating workbooks

In Microsoft Sentinel, we can use workbooks to define what we want to monitor and how we do it. Similar to alert rules, we have the option to use predefined templates or to create custom alerts. In contrast with alert rules, with workbooks, we create dashboards to monitor data in real time.

When the first edition of this book came out, there were 39 connectors available. This is another indicator of how fast Microsoft Sentinel is developing. At this moment, there are 114 templates available, and this list is very similar to the list of data connectors. Basically, there is at least one workbook template for each data connector. We can choose any template for the list displayed in the following screenshot:

Figure 8.16 – Microsoft Sentinel workbook templates

Each template will enable an additional dashboard that is customized to monitor a certain data source. In the following screenshot, we can see the dashboard for Azure activities:

Figure 8.17 – Azure Activity template

The story doesn't end here. With Microsoft Sentinel, we can leverage machine learning and embed intelligence in our security layer.

Using threat hunting and notebooks

In Microsoft Sentinel, with dashboards and alerts, we can look for anomalies and issues, but in modern IT, we need more. Cyber threats are becoming more and more sophisticated. Traditional methods of detecting issues and threats are not enough. By the time we detect issues, it may already be too late. We need to be proactive and look for possible issues and stop threats before they occur.

For threat hunting, there is a separate section in Microsoft Sentinel. It allows us to create custom queries, but also offers an extensive list of pre-created queries to help us start. Some of the queries for proactive threat hunting are high reverse DNS count, domains linked with the WannaCry ransomware, failed login attempts, hosts with new logins, and unusual logins, to name a few. A list of some of the available queries is shown in the following screenshot:

Figure 8.18 – Queries for threat hunting

We can also switch the view to live stream and use it to watch for certain events in real time. Live stream offers a graphical view of our hunting queries to help us monitor threats visually.

Another option in the hunting section is bookmarks. When exploring data and looking for threats, we may encounter events that we are not sure about. Some events may look innocent, but, in combination with other events, may prove very dangerous. Bookmarks allow us to save the results of some queries in order to revisit them later. We may want to check the same thing in the next few days and compare results, or maybe check some other logs that may give us more information.

Proactive hunting does not stop there. There is another section in Microsoft Sentinel, called notebooks. Microsoft Sentinel leverages data stores with the use of powerful queries and scaling to analyze data in massive volumes. Notebooks goes one step further and, with common APIs, allows the use of Jupyter Notebooks and Python. These tools extend what we can do with data stored in Microsoft Sentinel, allowing us to use a huge collection of libraries for machine learning, visualization, and complex analysis.

Again, we have some notebooks already available that we can start using right away. The list of available queries is shown in the following screenshot:

Figure 8.19 – Microsoft Sentinel notebooks

But it doesn't end there, Microsoft Sentinel provides additional capabilities when it comes to threat intelligence and threat detection. Let's now take a look at some of the advanced capabilities that Microsoft Sentinel provides.

Advanced threat detection

We already mentioned some of the threat detection options in Microsoft Sentinel, including built-in threat detection and NRT analytics rules. But there are also advanced options, which include User and entity behavior analytics (UEBA) and Multistage attacks (Fusion).

Every day, user actions generate a massive number of logs. Analyzing these logs can present an impossible task if we wanted to do them manually as we are looking at mountains of data, and it can be difficult to differentiate between malicious attempts and convert them into meaningful events. But we can't ignore them either as these can point us to threats caused by external entities (usually using compromised credentials) or even insiders (such as dissatisfied employees). UEBA provides the capability to analyze such logs and gain insight with the help of machine learning. Microsoft Sentinel can be connected to a number of data sources. Besides the obvious logs from Microsoft Azure and Azure Active Directory, other logs can be used, such as Syslog (Linux) or SecurityEvent (Windows) to name a few. Logs are aggregated and analyzed by Microsoft Sentinel in order to form a baseline (which represents normal behavior). Once a baseline is established, Microsoft Sentinel watches for any anomaly and flags it. Anomalies are also scored based on the probability of the user performing an action and the potential impact of the action. As machine learning is used, more data over more time will provide better results. With a bigger data sample to establish a baseline, results provided by UEBA will provide more accurate analyses and detect even the smallest incident.

Malicious attacks are getting more and more sophisticated and harder to discover. These attacks are often not direct or pointed at a single target, which makes them even harder to detect. We can define alerts and even actions for known attack scenarios, but what about emerging and unknown threats? Multistage attacks (Fusion) use a similar technic to UEBA and apply machine learning to this problem. Based on known attacks and data, Fusion uses algorithms to constantly learn and adapt. And not only the fact that it constantly learns and improves in order to detect new threats, but it can connect different invents that individually don't look suspicious but, when observed together, can present a significant threat.

It's important to mention that Microsoft Sentinel supports Bring Your Own Machine Learning (BYOML). With the use of Azure Databricks / Apache Spark and Jupyter Notebooks, we can have our own machine learning models and algorithms, analyze data, and publish results.

Microsoft Sentinel isn't just a tool with a limited number of options; it's also very customizable. There are many ways to adjust Microsoft Sentinel to your specific needs and requirements. Many external resources can be used or further adjusted.

Using community resources

The power of Microsoft Sentinel is extended by its community. There is a GitHub repository at https://github.com/Azure/Azure-Sentinel. Here, we can find many additional resources that we can use. Resources developed by the community offer new dashboards, hunting queries, exploration queries, playbooks for automated responses, and much, much more.

The community repository is a very useful collection of resources that we can use to extend Microsoft Sentinel with additional capabilities and increase security even further.

Summary

Microsoft Sentinel satisfies all the requirements for SIEM. Not only that, but it also brings additional tools to the table in the form of proactive threat hunting and using machine learning and predictive algorithms. With all the other resources we have covered, most aspects of modern security have been covered, from identity and governance over network and data protection, to monitoring health, detecting issues, and preventing threats.

But security does not end here. Almost every resource in Azure has some security options enabled. These options can help us go even further and improve security. With all the cybersecurity threats around today, we need to take every precaution available.

In the final chapter, we are going to discuss security best practices and how to use every security option to our advantage.

Questions

As we conclude, here is a list of questions for you to test your knowledge regarding this chapter's material. You will find the answers in the Assessments section of the Appendix:

1. Microsoft Sentinel is…

A. Security Event Management (SEM)

B. Security Information Management (SIM)

C. Security Information Event Management (SIEM)

2. Microsoft Sentinel stores data in…

A. Azure Storage

B. Azure SQL Database

C. A Log Analytics workspace

3. Which data connectors are supported in Microsoft Sentinel?

A. Microsoft data connectors

B. Cloud data connectors

C. A variety of data connectors from different vendors

4. Which query language is used in Microsoft Sentinel?

A. SQL

B. GraphQL

C. Kusto

5. Dashboards in Microsoft Sentinel are used for…

A. The visual detection of issues

B. Constant monitoring

C. Threat prevention

6. Rules and alerts in Microsoft Sentinel are used for…

A. The visual detection of issues

B. Constant monitoring

C. Threat prevention

7. Threat hunting is performed by….

A. Monitoring dashboards

B. Using Kusto queries

C. Analyzing data with Jupyter Notebooks