12.6. Presentation and Interpretation of Results

This section presents the results of the case study and discusses some of the conclusions that can be drawn from the data. The metrics defined in Section 12.5 are used.

Table 12.4 shows an overview of the drivers used for this case study. The number of services provided by the two drivers is similar (60 and 54), translating into a similar number of injections performed. The number of injections depends on the number of services, the number of parameters targeted, and the injection cases defined for each type. The drivers differ in the number of activated injections (i.e., injected errors actually executed), where the network driver has a higher activation rate (55% compared to 43%). The activation rate is a measure of how many experiments actually execute the error. The time required for executing the experiment depends on the number of experiments performed.

Table 12.5 details the results of the experiments for the serial port driver cerfio_serial.dll with respect to the OS services used by the driver. Only services resulting in failures are shown together with the number of error propagation observations, not their location. Table 12.6 shows the results for the Ethernet driver 91C111.dll, which shows more services leading to failure than cerfio_serial.dll. The rows in the tables are ordered according to the severity of the failures. For cerfio_serial.dll, it can be seen that no service leads to a crash of the system. However, for 91C111.dll FreeLibrary and LoadLibrary are both vulnerable services. 91C111.dll does not have as many cumulative Class 2 failures as cerfio_serial.dll, indicating that addition of a few robustness enhancing wrappers would remove all severe error propagation paths (Class 2 and Class 3 failures).

From the OS service point of view, Tables 12.7 and 12.8 show the used OS services, together with the OS service error exposure values. Alongside the number of failures for Class 1, the number of No Failure (NF) observations is also shown. The Class 2 and Class 3 failures affect all used OS service listed collectively. This effect is specific to the experiments conducted and does not translate into a general statement of OS behavior. For Class 2 failures, only the driver-specific test application was affected. Consequently, the OS service error exposures are calculated using only Class 1 failures. From these tables, one can find the services that are more exposed to propagating errors. For some services, the number of propagated errors is zero, indicating that the function was not affected by any of the injected errors (Class 1). On top of the tested OS services (Table 12.7), the correctness assertions are also included, which detect whether the correct information was received from the host computer. In this case, Correctness 1 failed 27 times, indicating that the first round of testing done in the application failed, where as the second round (Correctness 2) did not. This is not surprising given that each error is injected only once.

Tables 12.7 and 12.8 show that the results of the experiments “cluster” (i.e., an error in one service implies an error in another). This indicates dependencies across services, as well as nondependencies (or at least indication of weaker dependency). Some of these dependencies are expected, for instance that CreateFile affects ReadFile and WriteFile (Table 12.7). Some nondependencies are more unexpected, for instance that SetCommState is not affected by CreateFile. For both drivers, only one cluster appears, with 27 cases for seven services for the serial driver cerfio_serial.dll and 85 cases for three services for the Ethernet driver 91C111.dll.

For this case study, no OS service experienced failure as a result of propagating errors from more than one. This suggests that there is little correlation between failures in the OS services tested for both drivers, indicating the OS being able to limit error propagation in many cases.

Finally, Table 12.9 shows the resulting driver error diffusion values. The failure classes are presented separately as they have different failure impacts, with a Class 3 failure having higher impact than a Class 2, and so on. Table 12.9 shows that when considering error impact the network driver has more severe errors, whereas the serial driver has more Class 2 failures. Thus, these two classes of failures should be the first focus of the robustness enhancing activities. The network driver has overall more failures, but mainly of lesser impact.

Table 12.9. Results of injection experiments.

	Failure Class Distribution
Driver				Total
cerfio_serial.dll	0.460	0.022	0.0	0.482
91C111.dll	0.616	0.002	0.007	0.625