Design and Development (A3): SDL Activities and Best Practices 143
implements protective measures, such as code obfuscation, to pre-
vent reverse engineering. In this case, traditional debugging is not
possible. It should be noted that if the focus of debugging effort is
on the software interaction with an external component, the binary
may be all that is needed.
• Vulnerability scanning (black box). Automated vulnerability scan-
ning of operating system and application-level software involves use
of commercial or open-source scanning tools that observe execut-
ing software systems for behaviors associated with attack patterns
that target specific known vulnerabilities. Like virus scanners,
vulnerability scanners rely on a repository of “signatures,” in this
case indicating recognizable vulnerabilities. Like automated code
review tools, although many vulnerability scanners attempt to pro-
vide some mechanism for aggregating vulnerabilities, they are still
unable to detect complex vulnerabilities or vulnerabilities exposed
only as a result of unpredictable (combinations of) attack patterns.
In addition to signature-based scanning, most vulnerability scan-
ners attempt to simulate the reconnaissance attack patterns used by
attackers to “probe” software for exposed, exploitable vulnerabilities.
• Penetration testing (black box). The portion of security testing in
which the evaluators attempt to circumvent the security features of
a system. The evaluators might be assumed to use all systems design
and implementation documentation, which can include listings of
system source code, manuals, and circuit diagrams. The evaluators
work under the same conditions as are applied to ordinary users.
9–12
The test plan organizes the security testing process and outlines which
components of the software system are to be tested and what test proce-
dure is to be used on each one. The outline is more than just a list of high-
level tasks to be completed; it should also include which artifacts are to
be tested, what methodologies are to be used, and a general description of
the tests themselves, including prerequisites, setup, execution, and what
to look for in the test results. The risk analysis described previously is
typically used to prioritize the tests, since it is usually not possible, given
time and budget constraints, to test every component of the software.
Security test planning is an ongoing process, and the details of the test
process are fleshed out as additional information becomes available.
The developing organization may modify software because problems
have been uncovered and then send the software back to be retested.