To get audit data into the ACS database, you must enable auditing at the Windows operating system level on each ACS forwarder. You enable auditing through security policies. Security policy in Windows is found in three locations:

Windows Server 2003 Auditing Categories

Now we will closely examine the nine categories of auditing available in the Local Policy -> Audit Policy section of all three types of security polices (local, domain, and domain controller). Table 15.2 lists each category, a description of its effect, the audit settings (success or failure) in a default Windows installation, as well as the settings contained in the SecureDC.inf and SecureWS.inf security templates.

If your organization does not have more specific security needs (or if you just need a validated starting point to decide which auditing categories to enable and with what settings), we recommend importing the following templates:

• The SecureDC.inf security template to the Default Domain Controller Policy. Modify the Default Domain Controller Policy to enable success auditing for system events after importing the SecureDC.inf template.

• The SecureWS.inf security template to the Default Domain Security Policy.

The selection of category and audit settings shown in Table 15.2 is effective for collecting useful security audit data for most organizations.

Table 15.2. Window Server 2003 Audit Policies

Category	Description	Default Setting	“Secure” Templates
Account Logon Events	Audits each instance of a user logging on to or logging off from another computer in which this computer validates the account.	DC: Success Non-DC: None	Success, Failure
Account Management	Account management events including whether a user account or group is created, changed, or deleted; a user account is renamed, disabled, or enabled; and a password is set or changed.	DC: Success Non-DC: None	Success, Failure
Directory Service Access	Audits the event of a user accessing an Active Directory object that has its own system access control list (SACL) specified.	DC: Success Non-DC: N/A	Failure
Logon Events	Audits each instance of a user logging on to or logging off from a computer.	DC: Success Non-DC: None	Success, Failure
Object Access	Audits the event of a user accessing an object—for example, a file, folder, Registry key, printer, and so forth—that has its own system access control list (SACL) specified.	None	None
Policy Change	Audits every incident of a change to user rights assignment policies, audit policies, or trust policies.	DC: Success Non-DC: None	Success, Failure
Privilege Use	Audits each instance of a user exercising a user right, with the exception of Bypass traverse checking, Debug programs, Create a token object, Replace process level token, Generate security audits, and Back up or restore files and directories.	None	Failure
Process Tracking	Audits detailed tracking information for events such as program activation, process exit, handle duplication, and indirect object access	None	None
System Events	Audits when a user restarts or shuts down the computer or when an event occurs that affects either the system security or the security log, such as clearing the security log.	DC: Success Non-DC: None	None (Should be enabled for DCs)

In addition to deploying the appropriate audit settings through Domain Group Policy or Local Security Policy, we suggest you enforce minimum security log sizes for domain controllers and non-domain controllers. In a domain environment, you can easily configure the Security Settings -> Event Log -> Maximum security log size setting in the Domain and Domain Controller Security Policies. We recommend domain controllers have a maximum security log size of at least 163,840KB (160MB) and non-domain controllers at least 16,384KB. In a workgroup environment, apply these settings manually at each computer (Event Viewer -> Security -> Properties -> Log Size). Figure 15.2 shows the maximum security log size policy setting for domain controllers.

Figure 15.2. Setting the domain controller policy for the security event log size.

You may notice that the security auditing mechanisms of Windows are cleanly divided into domain controller and non-domain controller modes of operation. This is because most of the relevant security events you want to audit take place on the domain controller. Domain user accounts receive their authentication token from a domain controller and then use that token to access resources across the domain. Capturing the issue of that token on the domain controller is how any audit trail involving domain security will begin.

Note: Determining the Number of Security Events Generated

Joseph Chan has written a script that counts the number of security events generated per second on the local computer. (To run it against a remote computer, supply the computer name as an argument). The script will continue to run until you kill it using Ctrl+C. We suggest you pipe the output to a file. Here’s an example:

<LINELENGTH>90</LINELENGTH>

SecurityEventPerSecond.vbs >> NumberOfEventsGeneratedPerSec.csv

This command sends the output to a CSV file that you can load with Excel. This utility can be useful when you want to measure the incoming event rate before or without installing an ACS collector.

For your convenience, we include this script, SecurityEventPerSecond.vbs, on the CD accompanying this book.

Planning an audit policy in itself does not involve considering the ACS components of OpsMgr, because enabling auditing affects all computers in the domain, regardless of their potential status as forwarders. After you make your audit policy decisions and deploy audit policies to your managed computers, you are ready to prepare the ACS components that will forward, collect, and store the auditing events generated at each computer in accordance with the settings in the audit polices. Your deployment planning includes the following decision points for the ACS components:

Creating a Security Boundary

Beyond the SQL cluster/no SQL cluster issue, the preferred configuration for larger environments is separating the Collector and Database Components on different servers. Separating the processing load of these dissimilar components increases the capacity of the Audit Collection Services in the management group and provides security separation between the Collector and Database Components. Because the collector is always a management server, it is subject to administrative control by the OpsMgr administrators.

Figure 15.3 illustrates the OpsMgr components delivering Audit Collection Services in Operations Manager 2007. On the left side, you can see OpsMgr Agent Components on managed clients and servers, with the ACS forwarder enabled. While normal OpsMgr client-server communication occurs with the agents’ designated management server(s), ACS data is streamed directly from the forwarder to a collector, bypassing the regular OpsMgr management servers. On the right side of the diagram, you can see the ACS Audit Database Component and a representation of an external storage system (assuming long-term retention of audit data).

Figure 15.3. ACS topology for an organization with separation of audit control.

Notice in Figure 15.3 the vertical line labeled “Security Partition” separating the ACS database and archive storage system from the rest of the management group. This line represents a logical security boundary between the stored audit data and the audited management group.

It is not necessary, or in some environments even desirable, to install an OpsMgr Agent Component on the ACS database server in the same management group as the audited computers. For security considerations, you may decide to create a dedicated (even an “all in one server”) OpsMgr management group that monitors the ACS database computers in your organization. Using a separate management group prevents a careless or criminal user with elevated access to an OpsMgr configuration to run tasks on managed computers that completely compromise organizational security. Separating the ACS database server(s) (and any servers hosting long-term archive data from the organizational group they are data repositories for) can avoid this potential vulnerability.

How Many ACS Collectors Are Required and Their Placement

Another planning decision involves the number of ACS collector/database pairs to include in your OpsMgr deployment. Because each collector must have its own dedicated ACS database, there is always one ACS database (hosted on a standard or clustered SQL Server) for each collector. Whether you need more than one collector and database depends on the capacity of a single, given collector, as well as the bandwidth between the forwarder population(s) and their associated collector(s).

The maximum number of forwarders you can assign to the same collector will vary depending on the types of audit events selected in your audit policies. Assuming high-performance server hardware and sufficient database storage resources for the collectors and databases, here are estimated maximum loads of a single ACS collector/database pair:

• 150 domain controllers, or

• 3000 member servers, or

• 15,000 workstations

Most collectors have a mix of domain controller, member server, and workstation forwarders assigned to them. In that case, you can create weighted values for each type of ACS forwarder to estimate their aggregate load. We provide an example of creating weighted values later in this chapter, in the section “Managing ACS Database Size.”

A large number of forwarders cannot effectively share a low-bandwidth connection to a remote collector—for that scenario you might want to deploy several collectors, with each located in proximity to the largest forwarder populations. Figure 15.4 shows a possible ACS deployment with three collector/database pairs. Each collector is in a different physical location, with good network connectivity to its respective connected groups of ACS forwarders.

Figure 15.4. ACS topology with multiple collector-database pairs.

To estimate the bandwidth a particular forwarder requires to communicate with its collector, multiply the average size of a security audit event compressed for transmission (140 bytes) by the average number of security audit events in the Event log of the forwarder during a given period. For example, a moderately busy domain controller, with the desired security policy enabled, might generate 500,000 events in a 24-hour period. That equates to 70MB of network throughput per day, or about 1KB per second sustained bandwidth consumption.

Figure 15.4 illustrates each collector requiring its own dedicated ACS database and hosting SQL Server. You can implement a combination of clustered or non-clustered ACS SQL database servers, based on the requirements for each collector. This particular topology also features a shared long-term archive solution, although you can also architect dedicated archive solutions for each ACS database.

Tip: Sharing ACS Database Server Hardware

When you install ACS, you can select a named instance of SQL Server 2005 to connect to. To accommodate the one-to-one relationship between collectors and databases, you can use a single SQL Server 2005 box with named instances to create multiple ACS databases, provided it can support the additional load.

In Chapter 6, “Installing Operations Manager 2007,” we discuss the prerequisites and basic installation steps for deploying the ACS Database and Collector Components. This chapter picks up at that point. We have installed the ACS database on a separate server from the ACS collector, in the same manner as illustrated earlier in Figure 15.3.

If your organization does not include a separate auditing team—for example, if your OpsMgr deployment is in a smaller organization with a single IT group (or single individual!) and no one outside IT will use ACS—then no additional initial configuration of the ACS database is required. If, however, you are separating audit control from the IT group (typical in larger organizations), you must perform the following critical steps during the initial ACS database setup:

Optionally Create ACS Auditors Security Group

Installing the ACS components does not create any security groups in the domain or on the ACS collector and database computers. If you need to create an ACS auditors security group, you can create a local or domain security group based on your particular conditions. This security group will contain the user accounts of those individuals who access and run reports on the data in the ACS database.

Base your choice of whether to use a domain or local security group on the following criteria:

• “Ownership” of the security group

• Ease of monitoring the membership of the group to ensure it continues to contain only the desired members

If your auditing team also owns the ACS database server, using a local group makes it easier for that team to control membership of the ACS auditors’ security group, whereas a domain security group is easier for a central IT department to administer. Most likely, if you are not integrating ACS and OpsMgr reports, you will create a local security group on the SQL Reporting Server(s) dedicated to ACS. If you install ACS reports to the same instance of SQL Reporting Server used by the OpsMgr Reporting Component, a domain security group for ACS auditors is appropriate.

In our example, we will create a local security group on the ACS database computer (server name Fireball) and add domain user accounts as members to the group. If your ACS database is on a clustered SQL Server, create same-named local groups with the same memberships on each node of the cluster. We gave the local group on the ACS database server the name ACS Auditors. For our example, the ACS Auditors local group on Fireball has several user accounts from the Odyssey domain added as members. Figure 15.5 displays the local security group created on Fireball.

Figure 15.5. The ACS Auditors local group on the ACS database server contains domain user accounts as members.

In Figure 15.5, the users who are members of the local ACS Auditors security group, Dennis Cooper and Janet Valenti, represent Odyssey Corp’s employees charged with auditing the network. If those individuals are administrators of the ACS database server and the SQL Server application is running on it, they can guarantee unauthorized employees cannot access the ACS database—by verifying their user accounts remain the only members of the security group.

Granting ACS Database Permissions

Now we need to grant our ACS Auditors security group read-only rights to the ACS database. Read-only rights let those users access and run reports on the data in the ACS database without risk of modifying or deleting the collected security audit events. Perform this procedure using the SQL Server Management Studio application on the ACS database server. Open the SQL Server Management Studio and perform the following steps:

1. Navigate to Security -> Logins. Right-click Logins and select New Login to start the New Login Wizard.

2. Next to the Login name field, click the Search button.

3. On the Select Users or Group dialog box, click the Object Types button and add the Groups type of object. Click OK.

4. The From this location field should already contain the name of the ACS database server (the local machine and instance). If it does not, click the Locations button and select the local ACS database computer name. Click OK.

5. Enter the name of your local group (for example, ACS Auditors) and click OK. This returns you to the New Login Wizard. You should see the Login name field populated with your local ACS Auditors security group on the ACS database server.

6. In the Default database dropdown list, select the OperationsManagerAC database (assuming you used the default name during the installation) and click OK.

7. Navigate to Databases -> OperationsManagerAC -> Security -> Users. Right-click Users and select New User.

8. In the User name field, type a meaningful name, such as <Your Organization> ACS Auditors. (We use Odyssey ACS Auditors in our example.)

9. Click the selector button to the right of the Login name field. In the Select Login dialog box, click the Browse button.

10. In the Browse for Objects dialog box, locate the SQL login for the ACS database computer’s local ACS Auditors security group you just created. Check the box in the left column next to the local ACS Auditors security group and then click OK twice.

11. Back at the Database User – New dialog box, in the Database role membership area, check the box next to the db_datareader role and click OK.

Figure 15.6 shows the SQL Server Management Studio application after the described steps were used to create the ACS Auditors local group. We are viewing the properties of the Odyssey ACS Auditors database user. Notice the db_datareader role is the only selected role. Members of this role can run a SELECT statement against any table or view in the specified database.

Figure 15.6. Granting the ACS Auditors local group read-only rights to the ACS database.

ACS Database Default Install Does Not Allow User Access

Let’s examine the peculiar security settings of the OperationsManagerAC database. If you check the SQL user mapping performed during the ACS database installation on the SQL Server that hosts the database, you’ll notice that no users are listed in the security node of the OperationsManagerAC database. The effect of not including any user accounts is that normally privileged OpsMgr administrator and service accounts cannot access the ACS database. Only the computer account of the ACS collector has a SQL user mapping. Contrast these permissions with those on the OpsMgr operational database (which include the Management Server Action and SDK accounts) or those on the OpsMgr data warehouse database (which include the SDK and the Data Warehouse Writer and Reader accounts). You can see that the normal security environment in a management group does not extend to the ACS database; in fact, you see the ACS database is isolated from OpsMgr security permissions by design.

The computer account of the ACS Collector Component is the only account with permissions on the ACS database, of which it is the owner and runs using the SQL user account AdtServer. This matches the name of the ACS Collector service executable, which is AdtServer. AdtServer.exe runs on the ACS collector as the Operations Manager Audit Collection service. The AdtServer.exe process receives audit events over the network and writes them to the SQL database OperationsManagerAC using the computer account of the ACS Collector Component.

Audit Collection Service Reporting uses SQL 2005 Reporting Services. In the highest security scenario, which is Microsoft’s designed environment for using ACS, audit reports run on dedicated SQL 2005 Reporting Server instances that are not part of the OpsMgr 2007 management group. Optionally and alternatively, you can integrate ACS reports with the Reporting Server Component of the OpsMgr 2007 management group. The OpsMgr installation media includes the report models and an installation utility. Follow these steps to upload the ACS reports to your management group:

A fundamental decision is whether you will integrate ACS reports with your OpsMgr Reporting Server or will keep the ACS Reporting components completely separate from the OpsMgr management group. Microsoft designed ACS Reporting primarily to function in the “completely separate” model. Using this model, OpsMgr administrators have no access whatsoever to the collected audit data, report models, or templates uploaded to the SQL Reporting Server used by ACS. Auditing staff have complete ownership of, and exclusive access to, all the resources and data associated with collected security audit events.

This model is designed for larger and more security-conscious organizations. There are two other modes to deploy ACS reports; these other scenarios integrate ACS Reporting components with OpsMgr Reporting components to differing extents. The first alternative model installs the ACS reports to the OpsMgr Reporting Server but retains a separation between auditors and administrators. The second alternative model provides complete integration of ACS reports with OpsMgr reports but eliminates the security boundary. We will walk through these three different scenarios and discuss the procedures to implement them:

This scenario is only appropriate where there is no concern over Operations console users having the ability to run audit reports. In a smaller OpsMgr deployment or an organization without dedicated auditing staff, implementing this scenario lets you run audit reports from the Operations console in almost the same manner as regular OpsMgr reports. Microsoft’s design for ACS is that report data will only be accessed using SQL Report Manager and only by members of the ACS Auditors security group. This scenario implements the opposite of that design, namely that report data is accessed in the Operations console by any OpsMgr administrator or user who is a member of the OpsMgr Report Operators User or Administrator role.

In an integrated ACS reporting installation, if you attempt to run an audit report in the same fashion as other reports in the Operations console’s Reporting space, you will see the following error message:

An error has occurred during report processing.
Cannot create a connection to data source 'datasource1'

Reviewing the Application Event log of the Reporting Server, you will see that at that moment, the OpsMgr Data Reader account attempted to make a connection to the ACS database but permission was denied.

Because this scenario is not a supported configuration for ACS, if you choose to implement it, there may be various ways to circumvent default ACS security to make it work as desired. A simple way we discovered was to create the ACS Auditors local security group on the Reporting Server and then add the domain’s OpsMgr Data Reader Account to the ACS Auditors local security group. No changes to the default SQL data source are necessary when using this technique.

The stage is now set to begin collecting audit data from the managed computers. The final task in launching your organization’s audit collection components is to run the Enable Audit Collection task from the Operations console against the managed computers selected as ACS forwarders. By default, the service needed for an OpsMgr agent to be a forwarder (AdtAgent.exe) is installed but not enabled when the OpsMgr Agent Component is installed. Now that we have installed and configured the ACS collector and database, we must enable and configure the ACS Forwarder Component on those managed computers we identified in our audit policy to be ACS forwarders. Perform the following steps:

Figure 15.11. Application log connection events from the forwarder (left) and collector (right).

During the beta-testing phase for Operations Manager 2007, Joseph Chan of Microsoft provided a PowerShell script that enables ACS on all agents to the provided ACS collector. Neal Browne from SystemCenterForum has modified the script to take the display name of the group as an added parameter, so you can control which agents are enabled.

Table 15.3 lists the parameters used by the scripts.

For your convenience, we include these two scripts—ACSBulkEnableAllAgents.ps1 and ACSBulkEnableGroupDisplayName.ps1—on the CD accompanying this book. You can also view http://goteamshake.com/?p=34 for information on these scripts.

Now that you have deployed, configured, and enabled ACS, your organization enters the long-term operating phase of the auditing solution. Your primary technical goal is to set in motion a perpetual process that keeps the ACS components secure and functioning as desired by the auditing stakeholders. You achieve your business goal when appropriate auditing reports are reviewed in a timely and recurring fashion by cognizant internal network security staff or external auditing personnel.

Here are some questions to answer regarding processes you must implement to sustain a production ACS deployment:

Figure 15.12. Configuration table in the ACS database, where the retention period is set.

Importing the ACS reports into the management group using one of the integrated reporting scenarios creates a new folder in the Reporting space named Audit Reports. Inside the Audit Reports folder are 18 predefined reports and two report templates, included on the Operations Manager installation media. The audit report names are prefixed to sort alphabetically into six categories. The categories suggest scenarios in which specific predefined reports might be useful or applicable:

The two security audit report templates are cryptically named Audit and Audit5. The Audit template contains fields for up to 22 security audit event strings, basically 22 lines or elements of text that might appear in the Description area of a security audit event examined with the Windows Event Viewer. The Audit5 template provides for only the first five text elements (strings or lines of text) from the security audit event Description area. Use these criteria when selecting a template for the basis of a new custom report:

There are several differences in the behavior of the audit reports compared to the other predefined reports available in the OpsMgr Reporting space (remember that the data for security audit reports is coming from the ACS database rather than the Data Warehouse database like the other historical data viewed in the Reporting space):

Figure 15.13 is a screenshot of the Reporting pane in the OpsMgr Operations console, listing the audit reports and templates imported from the OpsMgr 2007 installation media.

Figure 15.13. Predefined reports and audit report templates, integrated with the Operations console.

Like all OpsMgr reports, one can schedule audit reports for publication to file shares for offline viewing, rather than viewing them live in the Operations console or with the web-based SQL Report Manager. Audit reports lend themselves in particular to scheduled and published report generation in these scenarios:

When scheduling publication of most security audit reports (also called creating a subscription in SQL Report Manager), you only need to specify the date range of the selected report. For those reports that also require the computer name parameter, there is a prompt for this when you schedule the report. Figure 15.14 shows the Report Parameters page while scheduling the Event Counts by Computer audit report. The scheduled report in this example will run against audit events from the computer Thunder (our Operational database server).

Figure 15.14. Scheduling an audit report to be generated and published for offline viewing.

Microsoft intended that ACS reporting use the SQL Report Manager to view ACS security audit data. Unless you implement the completely integrated ACS reporting scenario, the SQL Report Manager is used exclusively for live interaction with security event data collected in the ACS database.

Even if you implement the completely integrated ACS reporting scenario and use the Operations console to run audit reports routinely, there are still several ACS-related functions that can only be performed using SQL Report Manager. For example, deleting either a predefined or a custom audit report from the Reporting Server is accomplished only through Report Manager (select Audit Reports -> Show Details and then select the report to delete and click Delete).

You can drill down into the properties of any predefined or custom audit report using SQL Report Manager. In the properties of each report, you can view and modify details such as the report’s default parameters or security. After you have scheduled an audit report for automatic publishing to a file share, you can modify the settings of that subscription by returning to SQL Report Manager. Figure 15.15 shows the Subscriptions tab of the Unsuccessful Logon Attempts audit report. Selecting Edit here would allow you to modify the schedule, published file share location, and security credentials for generation of the report.

Figure 15.15. Modifying the properties of a previously scheduled audit report.

All new and custom ACS reports are created using SQL Report Builder, a component of the SQL Reporting Service. The first time you open an audit report for editing or start a custom report authoring session, the system downloads a 6.5MB Report Builder runtime module from the SQL Reporting Server to your computer. We will be taking a closer look at the Report Builder when we create a custom report in the “Policy Changes Scenario” section later in this chapter.

The primary tool for administering ACS is a command-line tool, AdtAdmin.exe. This tool runs locally on the collector at the server console or in a Remote Desktop session. There are four categories of functions for using AdtAdmin.exe:

We grouped the 12 AdtAdmin.exe commands into these categories in Table 15.4.

Creating and using forwarder groups is an advanced task, only useful in very large and distributed environments where there are multiple collectors with weighted priority values or you wish to operate against large numbers of forwarders at once. Many environments use AdtAdmin.exe primarily to confirm lists of ACS forwarder populations, confirm forwarder-to-collector assignments, and manage WMI queries that filter out some events before writing to the ACS database. We exercise these common AdtAdmin.exe tasks at the command line, as shown in Figure 15.16.

Figure 15.16. Performing an ACS Forwarder census and setting a WMI query filter.

Four command-line runs of the AdtAdmin.exe utility are shown in Figure 15.16. Here is an explanation of what is happening on each line:

We also wanted to include a list of the statistics returned with the command AdtAdmin.exe /stats. The output of this command can be redirected to a file (AdtAdmin.exe /stats > <filename>.csv) and the file opened in Excel as a Comma-Separated Value (CSV) text file. Viewed as a spreadsheet, there are 17 columns of data, with a row for each forwarder including a header row. Table 15.5 is a list of the contents of the header row, numbered from left to right as they appear in the spreadsheet.

From time to time, performance issues regarding the ACS collector may arise. These issues will manifest themselves in a lack of timeliness in writing security audit events to the ACS database or ceasing to collect events from forwarders. In an ideal ACS world, an unlimited number of forwarders could connect to a single ACS collector, and that collector would have limitless memory and network resources. In actuality, the capacity of a given collector is finite, and it is incumbent on the administrator of the ACS collector (the OpsMgr Administrator) to keep an eye on collector performance. Three Registry keys can be tuned to help performance. These keys work in conjunction with each other:

The first indication that a particular collector is nearing or exceeding its capacity limit is typically noticed when the ACS collector queue reaches the BackOffThreshold value. This is one of three parameters that control the behavior of the collector when it cannot write events to the database quickly enough. Memory on the collector is used for caching ACS events that need to be written to the ACS database. The BackOffThreshold value is a percent specifying how full the memory-based collector queue can become before the collector denies new connections from its forwarders. The default size of the collector queue is 262,144 events. With each event consuming an average 512 bytes, this means the collector queue occupies about 134MB of memory. Figure 15.17 shows the Registry Editor on the ACS collector open to the key where the threshold and queue levels are set (HKLMSystemCurrentControlSetServicesAdtServerParameters).

Figure 15.17. Performance-sensitive queue and threshold settings on the ACS Collector.

As exposed in the Registry of the collector in Figure 15.17, we see the BackOffThreshold is set to 75% of the MaximumQueueLength value. An additional performance-sensitive Registry key is highlighted in the figure, which is the DisconnectThreshold, also stated as a percent and by default set to 90% of the MaximumQueueLength. The settings of these three parameters work together to manage event insertion to the ACS database during capacity overload periods. When ACS is operating normally, the queue length should rarely reach the BackOffThreshold value, and it should never exceed the DisconnectThreshold (except during periods when the ACS database server is known to be unavailable, such as when the server is restarted after operating system updates have been loaded).

When the capacity of a given collector is completely exhausted, the sequence of events illustrated in Figure 15.18 will occur. Following the events clockwise in the figure, we describe what is happening at each step:

If you observe this sequence of events occurring frequently on a collector, that collector’s capacity has been exceeded and you need to modify the ACS environment to provide a credible and reliable audit collection service. To resolve collector performance issues, you can either reduce the volume of events written to the ACS database or add hardware to the ACS infrastructure.

To reduce the quantity of events, you can take one of the following approaches:

Hardware solutions to capacity issues include the following:

HKLMSYSTEMCurrentControlSetServicesAdtAgentParameters

A precursor condition, and one that may indicate collector performance issues, is increasing event latency—that is, an increase in the average time between event generation and collection. This is a measure of how quickly security audit events are captured to the ACS database after the events actually occur on the forwarder. A latency of between 1 and 2 seconds indicates a healthy forwarder-collector-database relationship. Although this latency may grow slightly as large numbers of forwarders are connected to a collector, if you keep an eye on this indicator, you will know when performance is beginning to degrade (hopefully in time to take corrective action before queue threshold events start to occur).

The Windows Performance Monitor is a great tool for observing ACS system health in real time. When you install the ACS Collector Component on an OpsMgr management server, two Performance Monitor objects are added: ACS Collector and ACS Collector Client. The ACS Collector object exposes 16 counters related to the functions of the ACS Collector server itself. The ACS Collector Client object adds just three counters—one of these is the average time between event generation and collection, a latency counter useful for assessing the health of a collector. Figure 15.19 shows the Performance Monitor running against the ACS collector.

Figure 15.19. Windows Performance Monitor with some relevant ACS counters loaded.

In the Windows Performance Monitor chart view shown in Figure 15.19, we have loaded several ACS Collector counters related to the collector itself (the bottom two rows in the chart legend). Notice the Database Queue % Full counter, which can help you keep tabs on that important measure of the ACS Collector queue before it exceeds any thresholds.

We also loaded a pair of ACS Collector Client counters: the Average time between event generation and collection in milliseconds counter and the Incoming Audits/sec counter. An instance of each counter is running against three individual forwarders (computers Pantheon, Quicksilver, and Thunder). This information is also available in a snapshot format by running AdtAdmin.exe /stats, but it can be more convenient and useful to watch a live representation of the data in the Performance Monitor.

You must closely manage the ACS database size on the storage system of the SQL Server hosting it. Actively tracking database growth can avoid a possible database shutdown or system crash, which will occur if the database consumes all disk space. Once forwarders begin to send security events to the collector, the ACS database will grow in size.

Assuming a constant number of forwarders connect to the collector and you are collecting a uniform daily quantity of security events, the database will grow at a constant rate until it reaches the ACS database retention period, which is 14 days by default. If nothing changes after that, beginning at the 15th day, the ACS database would remain at about the same size indefinitely.

In the real world, of course, conditions are often constantly changing. It is difficult to precisely manage the size of the ACS database, due to the large number of variables that can rapidly affect the quantity and type of events collected by ACS. You can, however, make some informed estimates of how much disk space your ACS database is going to need—based on the number and composition of forwarders that are planned to connect to a particular collector.

From reviewing the ACS database, we determined that immediately after a fresh installation of OpsMgr 2007 ACS components, the database size on disk was 132MB, with the used portion of the total database size being 8MB. Microsoft has also published the capacity of a single collector based on weighted values that correspond to the volume of security audit events expected by different types of ACS forwarders. These maximum capacities are:

This means we can assign a cost to each of these types of forwarders and interpolate mixes of the types of forwarders. We can assign workstations a value of 1 (15,000/15,000), member servers a value of 5 (15,000/3000), and domain controllers a value of 100 (15,000/150). We will use these weighted values in calculations that include all three types of forwarders.

Another key piece of data needed to create an ACS database size approximation is the average size (in MB) that a single forwarder contributes each day to database growth. We selected a base estimate of 8MB per day per workstation for this value, based on the data Microsoft published about its own experiences using ACS. We also have validated that member servers generate at least triple the quantity of events as workstations, and that domain controllers generate at least 100 times as many security events as workstations, so these weighted values appear reasonably accurate.

From this exercise, we developed the following formula to estimate the size of the ACS database:

(8 MB/day × (Number of Workstations) × Retention Days) +
(8 MB/day × (Number of Servers * 5) × Retention Days) +
(8 MB/day × (Number of Domain Controllers * 100) × Retention Days) +
8 MB (original base DB size) = ACS Database Size

Figure 15.20 displays a chart representing possible ACS database growth for given mixes of ACS forwarder types. At the top of the scale in our estimate, for a large environment with 3000 workstations, 600 servers, and 14 domain controllers, we can project an ACS database of about 840GB with a 14-day retention period. A large-scale real-world statistic comes from Microsoft’s own early use of ACS. Microsoft monitored the security of 5000 servers and 10,000 workstations, with a 30-day retention period, and it had about an 8TB (8000GB) ACS database. (We understand that Microsoft has since trimmed its retention period to 12 days.)

Figure 15.20. Estimated ACS database growth given different mixes of ACS Forwarder types.

If you experience ACS database growth that is beyond what the SQL Server hosting your database can support, you can take one of several approaches:

You can reduce the amount of data written to the database in the same manner as if you were experiencing overloading of a collector, namely collecting fewer events by changing audit policies or discarding unneeded events with WMI query filters. Reducing the number of days in the retention period is the primary way to quickly cut the size of the ACS database.

Earlier in the “Planning ACS Component Deployment” section of this chapter, we discussed the ability to cluster the SQL database server that hosts the ACS database. Doing so can provide uninterrupted audit collection services while one of the SQL cluster nodes is unavailable. However, the secondary management server that has the ACS Collector Component cannot be clustered, leaving a single point of failure in the audit collection architecture. If the collector is down temporarily, such as during a restart following the routine installation of updates, the forwarders remain in a disconnected state and audit collection is paused. If the ACS Collector component fails completely, such as a drive array crash, the organization stops collecting audit data while a new ACS Collector Component is installed and the ACS architecture realigned.

To avoid these situations involving downtime of the ACS Collector Component, install multiple ACS collector/database pairs in your organization and configure your ACS forwarders for failover operation. Figure 15.21 illustrates an organization that has deployed two ACS collector/database pairs and has enabled failover operation of its ACS forwarders.

Figure 15.21. Achieving highly available ACS Collector Components with agent failover.

The procedure to activate an ACS forwarder and assign that forwarder to a particular collector, covered previously in this chapter in the “Running the Enable Audit Collection Task” section, is not used when enabling failover operation. The forwarder locates collectors by checking the Registry on the local agent computer. A list of ACS collectors is maintained at this Registry key on each forwarder:

HKLMSOFTWAREPoliciesMicrosoftAdtAgentParametersAdtServers

When the forwarder starts up, and when it is attempting to reconnect to a collector after being disconnected, it looks up the list of collector names at that Registry key and connects to the first one it can establish a connection with. The value at that key is of the REG_MULTI_SZ type, meaning that multiple string elements can exist, each on a different line of the value. When you invoke the Enable Audit Collection task in the Operations console, you can only enter a single collector name (a single string value), and the task running on the forwarder will overwrite whatever was at that Registry key with the single string value.

To implement ACS agent failover support in your organization, you must deploy one or more additional ACS collector/database pairs. You still run the Enable Audit Collection Task in the Operations console to enable an ACS forwarder initially—specify the name of that forwarder’s assigned primary collector when you run the task. Then you return to each ACS forwarder and edit the Registry key, adding the alternate forwarder names on the second and subsequent lines of the REG_MULTI_SZ value.

The VBScript command shell.regwrite, often used to automate Registry changes, does not handle multistring values, so you will have to make the change manually on computers that are to participate in the ACS agent failover scheme. The AdtServer Registry key, configured on a given forwarder as desired for a particular set of primary and alternate collectors, could be exported to a .REG file using the Regedit utility, and that .REG file merged into forwarder registries.

Once the AdtServer Registry key is configured with multiple collector names for agent failover support, be careful not to run the Enable Audit Collection Task in the Operations console again, because doing so overwrites the multiple collector names with the one name specified when running the task!

To force failover of forwarders to their alternate ACS collector(s), you can stop the Operations Manager Audit Collection service (AdtServer.exe) on the primary collector. Once a forwarder connects to an alternate collector, it will remain connected until disconnected for any reason (such as running AdtAdmin.exe /disconnect on the alternate collector). After each disconnection event, the forwarder again processes the list of collectors found at the Registry key. If the primary collector is available, the forwarder will connect to it and resume normal communication.

There may be situations where you want to audit computers that are not in the domain of the management group, or are not in domain(s) that trust the management group domain. Specifically, you may have computers in workgroups and in untrusted domains that need to participate in ACS.

To provide security auditing services to a computer that does not trust the collector, first enable certificate-based mutual authentication of the OpsMgr Agent Component as discussed in Chapter 11, “Securing Operations Manager 2007.” Then follow these steps to enable ACS: