Verifying and Validating Proper Configuration and the Implementation of Security Controls and Countermeasures

Auditing security controls across the IT infrastructure involves testing the controls or countermeasures using available documents, interviews, and personal observation.

This section provides an overview of testing and validating controls based upon NIST SP800-53A, which provides an approach to assessing security controls. Regardless of the exact methods used, however, the principles are the same.

Each control to be tested should have an accompanying assessment objective. The objective provides the foundation or high-level statement to determine the effectiveness of the control. Based on this, one or more assessment objectives are validated using a specific method. Such methods include examination, interviews, and testing. Using these methods against particular assessment objects will produce the results, which is a determination of the effectiveness of the controls. The assessment objects vary and include different types of elements. Three broad categories of objects include the following:

• Specification objects—These include documents such as policies, procedures, plans, and architectural designs.

• Mechanism objects—These are the specific hardware and software countermeasures installed and configured.

• Activity objects—These are the security-related actions involving IT personnel.

The effort required to assess controls will vary not just across the objectives. The auditor should also consider impact levels, or the sensitivity and importance of the information systems. The effort is directly related to the depth and breadth of the assessment. NIST’s assessment framework uses the terms depth and coverage and also defines their associated values. A summary of these definitions is as follows:

• Depth—This addresses the thoroughness and level of detail in the examination, interview, and testing process.

• Coverage—This addresses the breadth of the examination, interview, and testing process of objects.

Depth and coverage each have three different values. The values that address depth are generalized, focused, or detailed. The values that address coverage are representative, specific, and comprehensive. Representative coverage makes use of a limited sample of assessment objects. Further, the goal is to determine that a security control is put in place and that there aren’t obvious faults. More specific coverage builds on this by increasing the scope to achieve greater confidence that the control is not only put in place correctly with no obvious faults, but it is also operating as intended. Finally, comprehensive coverage uses a much larger sample of assessment objects to achieve the results of representative and specific coverage. It also ensures the control is operating on an ongoing basis that is continually improved.

The varying levels of depth have the same relative expectations. This includes making sure controls are put in place and free of obvious errors. It also includes making sure the controls are consistently operating as intended and are supported by continual improvement. Interviews and document reviews at a general level include high-level discussions and examination. A more focused assessment asks questions in greater depth and requires a more detailed analysis of documents. Finally, a detailed assessment includes asking deep, probing questions and performing thorough analysis of documents across a greater body of evidence.

In addition to interviews and document reviews, testing depth uses methodologies that require varying degrees of knowledge about the environment being tested. This ranges from having no knowledge of the infrastructure or implementation of a control to having considerable and extensive knowledge of both the infrastructure and details about the control.

Based on the tests of each control, an unbiased and factual determination is made as to the effectiveness of the control. The control should either satisfy or not satisfy the expected state. In some situations, the auditor will not be able to determine how effective a control is because of lack of information or an inability to test. In situations where the objectives of the control are not fulfilled, the auditor needs to understand and document how the control differs from what is expected. In addition, the auditor should note how these findings affect confidentiality, integrity, and availability.