13.6. Benchmarking Intrusion Response Systems

It is important to benchmark any IRS using quantifiable metrics. This is a nascent field within IRS design and development and one which needs significant work to get to maturity. Hence, this section is based around a suggested course of action for future development and an example from an existing IRS, ADEPTS. The metrics should capture the two essential goals of IRSs: to provide gracefully degraded functionality in the presence of attacks and to make a system more robust to future attacks. These two notions are addressed respectively by the metrics survivability and vulnerability.

One commonly accepted definition of survivability is the capacity of a system to provide essential services in the face of intrusions [40, 41]. The challenge with this definition is how to define essential services: Is this by the different categories of users for the different services, or by business criticality, or by some other measure? Also, the question arises if there exists a minimum essential service level that can be guaranteed. In Jha et al. [42], the authors inject errors into a network specification and visualize effects in the form of scenario graphs. Model checking is used to verify if states that violate certain temporal properties can be reached. Hiltunen et al. [18] present Cactus, which is a framework for constructing highly customizable and dynamically adaptable middleware services for networked systems. The fine-grained customization allows customized trade-offs between QoS attributes, including performance, reliability, and survivability, while the dynamic adaptation allows services to change behavior at runtime as a reaction to incoming intrusions.

To start, consider a simple combinatorial model for survivability. Let us define G as the overall goal of the system (e.g., sell products or services on the Internet), which is accomplished through several subgoals G_i (e.g., the different transactions that are possible on the system), i = 1,...,N. Each subgoal G_i is given a weight W_i indicating its importance in the achievement of the system goal G,. This can be estimated by the number of users who reach the subgoal as a fraction of the total number of users, a fraction of the usage of the subgoal, or a quantity defined by the system owner. Each subgoal G_i is decomposed into a conjunction of sets of services , such that each set of services must be functional for goal G_i to be reached. Each such set can be further decomposed to be a disjunction of basic services , such that any service of this set being functional causes the set to be functional. Let p_x denote the probability that a service X is affected by a disruption and cannot be used, and P_Y denote the probability that a goal y cannot be reached. Then, P_Gi = Max (Min (ps_ijk), over all k = 1,...,N_ij), over all . The survivability is given by 1 – P_G.

To apply this formulation, we will have to decompose a goal into the services and estimate the probability of a service being nonfunctional. The former can be deduced from a Service Net (network indicating interactions between services during normal operation) through a training phase when the transaction corresponding to a particular subgoal G_i is executed and the service interactions observed. We may use techniques from software reliability engineering of path testing [43] to determine the conjunction and disjunction of services.

To illustrate the concept, let us consider an example of its application as shown by Foo et al. in ADEPTS[26]. Figure 13.2 depicts the test bed that is used for experiments on ADEPTS. The payload system mimics an e-commerce web store, which has two Apache web servers running web store applications based on Cube-Cart [44] and are written in the PHP scripting language. In the backend, there is a MySQL database that stores all the store’s information, which includes products inventory, products description, customer accounts, and order history. There are two other organizations with which the web store interacts: a bank and a warehouse. The bank is a home-grown application that verifies credit card requests from the web store. The warehouse is also a home-grown application that takes shipping requests from the web store, checks inventory, applies charges on the customers’ credit card accounts, and ships the products. The clients submit transactions to the web store through a browser. Some important transactions are given in Table 13.1.

Figure 13.2. Layout of e-commerce test bed for the experiments on ADEPTS.

[View full size image]

There are certain security goals for the system, the complement of which are specified in Table 13.2, along with the weights. Thus, adding the word “prevent” before each gives the goal. The attached weights to the transactions and security goals are used for survivability computation as discussed below.

The authors define survivability based on the high-level transactions and security goals. Thus, the metric shows the effect of ADEPTS on the high-level functioning of the e-commerce system:

When a transaction became unavailable or the security goal is violated, the survivability drops by its corresponding weight, which was given in Tables 13.1 and Table 13.2. Transactions become unavailable due to ADEPTS responses, such as rebooting a host or due to attacks. Security goals may be violated due to the successful execution of an attack step or an erroneous response action. If a security goal is violated multiple times during an attack, then each violation causes a decrease in the survivability.

The survivability metric considers the state of the system at the present time and does not consider the resilience of the system to future disruptions. This is an important measure and is captured by the vulnerability metric. The basic idea of this metric is to fit a temporal distribution for the probability that a given goal in a multistage attack is reached. This curve is analogous to unreliability curves seen in traditional fault-tolerant systems. To get the vulnerability at a point in time T, we aggregate the individual unreliability curves using some structure as an attack graph and map the nodes to the services that are affected. Then an analysis similar to the survivability analysis above is performed. Note that the different curves are not independent. Given the Service Net with times for interaction between services, the edges in the attack graph will also have time for the delay between a lower-level goal and a higher-level goal being achieved. We believe the dependence introduces substantial complexity in the analysis and requires further investigation.