Security Information and Event Management (SIEM) combines two solutions that were previously separate, Security Information Management (SIM) and Security Event Management (SEM).
We have already mentioned that large organizations rely on SIEM solutions. And Microsoft's SIEM solution for the cloud is Microsoft Sentinel. But let's first take a step back and discuss what SIEM is and what functionalities it should have.
We will be covering the following topics in this chapter:
Many security compliance standards require long-term storage, where security-related logs should be kept for long periods of time. This varies from one compliance standard to another and can be up to many years. This is where SIM comes into the picture: long-term storage where all security-related logs are stored for analysis and reports.
When we speak of SEM, we tend to be talking about live data streaming rather than long-term event tracking. SEM's focus is on real-time monitoring; it aims to correlate events using notifications and dashboards. When we combine these two, we have SIEM, which tries to live stream all security-related logs and keep them for the long term. With this approach, we have a real-time monitoring and reporting tool in one solution.
When discussing the functionalities required, we have a few checkboxes that SIEM must tick:
To summarize, SIEM should have the ability to receive different data types in real time, provide meaning to received data, and store it for an extended period of time. Received data is then analyzed to find patterns, detect anomalies, and help us prevent or stop security incidents.
So, let's see how Microsoft Sentinel addresses these requirements.
Microsoft Sentinel is Microsoft's SIEM solution in the cloud. As cloud computing continues to revolutionize how we do IT, SIEM must evolve to address the new challenges that the changes to IT create. Microsoft Sentinel is a scalable cloud solution that offers intelligent security and threat analytics. On top of that, Microsoft Sentinel provides threat visibility and alerting, along with proactive threat hunting and responses.
So, if we look carefully, all the checkboxes for SIEM are ticked.
The pricing model for Microsoft Sentinel comes with two options:
Important Note
At the time of writing, the price in the Pay-As-You-Go model was $2.60 per ingested GB.
The following diagram shows the capacity reservation prices for Microsoft Sentinel at the time of writing:
If ingested data exceeds the reservation limit, further billing is done based on the Pay-As-You-Go model. For example, if we have a reserved capacity of 100 GB and we ingest 112 GB, we will pay the tier price for the reserved capacity up to 100 GB and will also pay for the additional 12 GB for the data that exceeds the reservation.
It's interesting to reflect on pricing presented in the first edition of this book and how it changed in 2 years. The price decreased in the region of 5%, but I find capacity options more interesting. 2 years ago, the maximum capacity was 500 GB per day, while today we have a capacity of up to 5,000 GB per day. This perfectly paints the picture of constant data growth and how data is becoming bigger and bigger right before our eyes.
When enabling Microsoft Sentinel, we need to define the Log Analytics workspace that will be used for storing data. We can either create a new Log Analytics workspace or use an existing one.
Microsoft Sentinel uses Log Analytics for storing data. The price for Microsoft Sentinel does not include charges for Log Analytics.
Important Note
An additional charge for Log Analytics will be incurred for ingested data. Information about pricing can be found at https://azure.microsoft.com/en-us/pricing/details/monitor/.
In the following screenshot, we can see all the pricing options for Microsoft Sentinel at the time of writing:
Data retention is configured in Log Analytics Workspace. Log Analytics Workspace default retention is 30 days, but it can be increased to a period of up to 2 years. However, keep in mind that changing the retention period will create additional costs as this can increase the amount of data significantly.
Important Note
Microsoft Sentinel customers can increase data retention to 90 days for free. This only applies to a Log Analytics workspace connected to Microsoft Sentinel.
Now, let's see how Microsoft Sentinel does everything that SIEM should be able to do. We will analyze all the requirements one by one and see how they are satisfied by Microsoft Sentinel.
Let's start with data connectors and retention.
One of SIEM's requirements is data aggregation. However, data aggregation doesn't just mean collecting data, but also the ability to collect data from multiple sources. Microsoft Sentinel does this really well and has many integrated connectors. We can see how much attention Microsoft is paying to this part when comparing the number of available data connectors between the two editions of this book. In the first edition, it was mentioned that 32 data connectors were available. Today, we have 120 of them, an increase of over 350% in 2 years. A lot of the connectors are for different Microsoft sources, such as Azure Active Directory, Azure Active Directory Identity Protection, Microsoft Defender for Cloud, Microsoft Cloud App Security, or Office 365, to name but a few. However, there is also a number of connectors for data sources outside the Microsoft ecosystem, such as Amazon Web Services, Barracuda Firewalls, Cisco, Citrix, and Palo Alto Networks.
Important Note
For more information on connectors, you can refer to the following link: https://docs.microsoft.com/en-us/azure/sentinel/connect-data-sources.
The data connectors page is shown in the following screenshot:
All data connectors include step-by-step instructions explaining how to configure data sources to send data. It's important to mention that all the data is stored in Log Analytics. Instructions vary from data source to data source. Most Microsoft data sources can be added by just enabling a connection from one service to another. Other data sources require the installation of an agent or editing of the endpoint's configuration.
There are many default connectors in Microsoft Sentinel. Besides the obvious Microsoft connectors with services in Azure and Office 365, we have many connectors for on-premises services as well. But it doesn't stop there, and many other connectors are available, including Amazon Web Services, Barracuda, Cisco, Palo Alto, F5, and Symantec.
Once the data is imported into Log Analytics and is ready to be used in Microsoft Sentinel, that is when the real work starts.
After the data is gathered, the next step is to display data using various dashboards. Dashboards visually present data using Key Performance Indicators (KPIs), metrics, and key data points in order to monitor security events. Presenting data visually helps set up baseline patterns and detect anomalies.
The following screenshot shows events and alerts over time:
In this screenshot, we can see how the baseline is established. There is a similar number of events over time. Any sudden increase or decrease would be an anomaly that we would need to investigate.
The Events and alerts over time dashboard uses metrics to display data, but we can also use KPIs to create different types of dashboards. The following diagram shows anomalies in the data source:
These two examples represent default dashboards that are available when Microsoft Sentinel is enabled. We can also create custom dashboards, based on the requirements and the KPIs defined.
However, this is only the first step in detecting something that's out of the ordinary. We need additional steps to automate the process.
The only problem with dashboards is that they are useful if someone is watching them. Data is displayed visually, and we can detect issues only if we are monitoring dashboards at all times. But what happens when no one is watching? For these situations, we define rules and alerts.
Using rules, we can define a baseline and send notifications (or even automate responses) when any type of anomaly appears. In Microsoft Sentinel, we can create custom rules on the Analytics blade, with three types of rules available:
To create a custom query rule, the following steps are required:
Optionally, we can add a description. Adding a description is recommended because it can help us track down rules and detect their purpose. An example is shown in the following screenshot:
The query used in this example is tracking resources on the account that were logged in the last 24 hours, which you can see here:
let GetAllHostsbyAccount = (v_Account_Name:string){
SigninLogs
| extend v_Account_Name = case(
v_Account_Name has '@', tostring(split(v_Account_Name, '@')[0]),
v_Account_Name has '', tostring(split(v_Account_Name, '')[1]),
v_Account_Name
)
| where UserPrincipalName contains v_Account_Name
| extend RemoteHost = tolower(tostring(parsejson(DeviceDetail.['displayName'])))
| extend OS = DeviceDetail.operatingSystem, Browser = DeviceDetail.browser
| extend StatusCode = tostring(Status.errorCode), StatusDetails = tostring(Status.additionalDetails)
| extend State = tostring(LocationDetails.state), City = tostring(LocationDetails.city)
| extend info = pack('UserDisplayName', UserDisplayName, 'UserPrincipalName', UserPrincipalName, 'AppDisplayName', AppDisplayName, 'ClientAppUsed', ClientAppUsed, 'Browser', tostring(Browser), 'IPAddress', IPAddress, 'ResultType', ResultType, 'ResultDescription', ResultDescription, 'Location', Location, 'State', State, 'City', City, 'StatusCode', StatusCode, 'StatusDetails', StatusDetails)
| summarize min(TimeGenerated), max(TimeGenerated), Host_Aux_info = makeset(info) by RemoteHost, tostring(OS)
| project min_TimeGenerated, max_TimeGenerated, RemoteHost, OS, Host_Aux_info
| top 10 by min_TimeGenerated desc nulls last
| project-rename Host_UnstructuredName=RemoteHost, Host_OSVersion=OS
};
// change <Name> value below
GetAllHostsbyAccount('<Name>')
The following screenshot shows an example of a schedule and a threshold:
Several optional settings allow us to enhance alerts with entity mapping, custom details, or alert details. We can also select event grouping and choose whether we want a query to stop running if an alert is generated.
To have an automated response, we need to create playbooks. Let's take a look at how playbooks are created in Microsoft Sentinel.
Under Automation in Microsoft Sentinel, we create playbooks that can provide an automated response if an alert is triggered. Responses can vary from blocking and preventing threats to sending notifications, depending on the severity.
We can either create a playbook from scratch or select an existing template. Let's take a look at the templates and select Block AAD user – Alert, as in the following example:
If we select a template and choose to create a playbook, it will take us to a new blade. We need to select a subscription, resource group, region, and playbook name. An example is shown in the following screenshot:
Optionally, we can select to send diagnostics logs to Log Analytics and configure different integrations.
Next, we can see connectors that will be created for the playbook in question, as shown:
Finally, we review the settings and create a playbook.
Note that the connection created needs to be authorized once deployment is done. And now we have a way to automatically respond to threats and incidents.
But in modern cybersecurity, this may not be enough. Let's look at other options in Microsoft Sentinel.
In Microsoft Sentinel, we can use workbooks to define what we want to monitor and how we do it. Similar to alert rules, we have the option to use predefined templates or to create custom alerts. In contrast with alert rules, with workbooks, we create dashboards to monitor data in real time.
When the first edition of this book came out, there were 39 connectors available. This is another indicator of how fast Microsoft Sentinel is developing. At this moment, there are 114 templates available, and this list is very similar to the list of data connectors. Basically, there is at least one workbook template for each data connector. We can choose any template for the list displayed in the following screenshot:
Each template will enable an additional dashboard that is customized to monitor a certain data source. In the following screenshot, we can see the dashboard for Azure activities:
The story doesn't end here. With Microsoft Sentinel, we can leverage machine learning and embed intelligence in our security layer.
In Microsoft Sentinel, with dashboards and alerts, we can look for anomalies and issues, but in modern IT, we need more. Cyber threats are becoming more and more sophisticated. Traditional methods of detecting issues and threats are not enough. By the time we detect issues, it may already be too late. We need to be proactive and look for possible issues and stop threats before they occur.
For threat hunting, there is a separate section in Microsoft Sentinel. It allows us to create custom queries, but also offers an extensive list of pre-created queries to help us start. Some of the queries for proactive threat hunting are high reverse DNS count, domains linked with the WannaCry ransomware, failed login attempts, hosts with new logins, and unusual logins, to name a few. A list of some of the available queries is shown in the following screenshot:
We can also switch the view to live stream and use it to watch for certain events in real time. Live stream offers a graphical view of our hunting queries to help us monitor threats visually.
Another option in the hunting section is bookmarks. When exploring data and looking for threats, we may encounter events that we are not sure about. Some events may look innocent, but, in combination with other events, may prove very dangerous. Bookmarks allow us to save the results of some queries in order to revisit them later. We may want to check the same thing in the next few days and compare results, or maybe check some other logs that may give us more information.
Proactive hunting does not stop there. There is another section in Microsoft Sentinel, called notebooks. Microsoft Sentinel leverages data stores with the use of powerful queries and scaling to analyze data in massive volumes. Notebooks goes one step further and, with common APIs, allows the use of Jupyter Notebooks and Python. These tools extend what we can do with data stored in Microsoft Sentinel, allowing us to use a huge collection of libraries for machine learning, visualization, and complex analysis.
Again, we have some notebooks already available that we can start using right away. The list of available queries is shown in the following screenshot:
But it doesn't end there, Microsoft Sentinel provides additional capabilities when it comes to threat intelligence and threat detection. Let's now take a look at some of the advanced capabilities that Microsoft Sentinel provides.
We already mentioned some of the threat detection options in Microsoft Sentinel, including built-in threat detection and NRT analytics rules. But there are also advanced options, which include User and entity behavior analytics (UEBA) and Multistage attacks (Fusion).
Every day, user actions generate a massive number of logs. Analyzing these logs can present an impossible task if we wanted to do them manually as we are looking at mountains of data, and it can be difficult to differentiate between malicious attempts and convert them into meaningful events. But we can't ignore them either as these can point us to threats caused by external entities (usually using compromised credentials) or even insiders (such as dissatisfied employees). UEBA provides the capability to analyze such logs and gain insight with the help of machine learning. Microsoft Sentinel can be connected to a number of data sources. Besides the obvious logs from Microsoft Azure and Azure Active Directory, other logs can be used, such as Syslog (Linux) or SecurityEvent (Windows) to name a few. Logs are aggregated and analyzed by Microsoft Sentinel in order to form a baseline (which represents normal behavior). Once a baseline is established, Microsoft Sentinel watches for any anomaly and flags it. Anomalies are also scored based on the probability of the user performing an action and the potential impact of the action. As machine learning is used, more data over more time will provide better results. With a bigger data sample to establish a baseline, results provided by UEBA will provide more accurate analyses and detect even the smallest incident.
Malicious attacks are getting more and more sophisticated and harder to discover. These attacks are often not direct or pointed at a single target, which makes them even harder to detect. We can define alerts and even actions for known attack scenarios, but what about emerging and unknown threats? Multistage attacks (Fusion) use a similar technic to UEBA and apply machine learning to this problem. Based on known attacks and data, Fusion uses algorithms to constantly learn and adapt. And not only the fact that it constantly learns and improves in order to detect new threats, but it can connect different invents that individually don't look suspicious but, when observed together, can present a significant threat.
It's important to mention that Microsoft Sentinel supports Bring Your Own Machine Learning (BYOML). With the use of Azure Databricks / Apache Spark and Jupyter Notebooks, we can have our own machine learning models and algorithms, analyze data, and publish results.
Microsoft Sentinel isn't just a tool with a limited number of options; it's also very customizable. There are many ways to adjust Microsoft Sentinel to your specific needs and requirements. Many external resources can be used or further adjusted.
The power of Microsoft Sentinel is extended by its community. There is a GitHub repository at https://github.com/Azure/Azure-Sentinel. Here, we can find many additional resources that we can use. Resources developed by the community offer new dashboards, hunting queries, exploration queries, playbooks for automated responses, and much, much more.
The community repository is a very useful collection of resources that we can use to extend Microsoft Sentinel with additional capabilities and increase security even further.
Microsoft Sentinel satisfies all the requirements for SIEM. Not only that, but it also brings additional tools to the table in the form of proactive threat hunting and using machine learning and predictive algorithms. With all the other resources we have covered, most aspects of modern security have been covered, from identity and governance over network and data protection, to monitoring health, detecting issues, and preventing threats.
But security does not end here. Almost every resource in Azure has some security options enabled. These options can help us go even further and improve security. With all the cybersecurity threats around today, we need to take every precaution available.
In the final chapter, we are going to discuss security best practices and how to use every security option to our advantage.
As we conclude, here is a list of questions for you to test your knowledge regarding this chapter's material. You will find the answers in the Assessments section of the Appendix:
A. Security Event Management (SEM)
B. Security Information Management (SIM)
C. Security Information Event Management (SIEM)
A. Azure Storage
B. Azure SQL Database
C. A Log Analytics workspace
A. Microsoft data connectors
B. Cloud data connectors
C. A variety of data connectors from different vendors
A. SQL
B. GraphQL
C. Kusto
A. The visual detection of issues
B. Constant monitoring
C. Threat prevention
A. The visual detection of issues
B. Constant monitoring
C. Threat prevention
A. Monitoring dashboards
B. Using Kusto queries
C. Analyzing data with Jupyter Notebooks