310 Core Software Security
to deliver quality results. Alan Paller once casually suggested to one of the
authors that there were not more than 1500 skilled penetration testers
extant. We don’t know the actual number, but there are not enough
highly skilled penetration testers to deliver all the work that is needed.
This situation will probably be true for some time. Due to the scarcity,
we suggest that attack and penetration testing be reserved for critical com-
ponents, and major releases that are expected to be under severe attack.
If your organization has sufficient attack and penetration resources,
the skilled human element is the strongest testing capability in security.
Everything that can be tested probably should be tested. However, we
have seen too many findings reports where the tester did not have this
kind of skill, did not take time to understand the target of the test, ran
the default tests, and reported hundreds of vulnerabilities. These sorts of
tests help no one. Development teams may look at the first few vulner-
abilities, declare them false positive, and stop looking. This is a classic,
typical response to a report filled with possible vulnerabilities rather than
real issues. Generally, in these cases, the attack test was not tuned and
configured to the target, and perhaps the target was not properly config-
ured as it would be when deployed. In our experience, this is a big waste
of everyone’s time.
Instead, focus your highly skilled resources or dollars on the most wor-
thy targets. Critical code that must not fail can benefit greatly from an
A&P test. And a strong return on investment can be made before major
releases or after major revisions. This is where we suggest the most benefit
can be gained from skilled attack and penetration testing.
Because an attack and penetration test can take considerable time to
complete, the rate of code change must be considered when applying
this intensive type of test. If the rate of change (update) is faster than the
length of time to test the system, vulnerabilities may be introduced before
the test even completes. These two factors must be weighed in order to
get the most useful results. Generally, even if updating occurs every day,
these will not be major releases, and certainly not major revisions. Hence,
testing at the larger code inflections can be a better investment.
What is critical code? We have seen numerous definitions of “critical”:
• The highest-revenue system
• The most attacked system
• The largest system
Applying the SDL Framework to the Real World 311
• The system with the biggest investment
• The most strategic system
• The most regulated system
• The highest-risk system
• The system handling the most sensitive data
Each one of these definitions can be blown apart easily with examples
of the others. A practical approach is to let business leaders or other
organizational leaders decide which systems are critical. Multiple factors
may be taken into account. None of the definitions above are mutu-
ally exclusive; different factors may add weight to the criticality of a
system. We suggest an open approach. A larger net, if the organization
can afford it, is probably better in the long run. An organization doesn’t
want to miss an important system simply because it failed any single
factor for criticality.
9.4.4 Independent Testing
There may be situations where it’s advantageous to apply third-party secu-
rity testing. If customers for a product are particularly security-sensitive,
they may demand a third-party verification of the security health of the
system before purchasing.
In the case of demonstrable customer demand for security verification,
one successful approach that we have used is to have third-party testing be
accounted for as a direct cost of goods sold. When systems can’t be sold to
many customers without third-party verification, third-party verification
is considered a part of the cost of building the system for those customers.
One report can typically be used for many customers.
Indeed, sometimes there are advantages to getting an independent
view of the system. As in all human endeavors, if the evaluators are too
close to the system, they may miss important factors. Applying some
independent analysis will focus fresh eyes on the problems.
“Independent” doesn’t necessarily mean outside the organization
entirely. We have had success simply bringing in a security architect who
hadn’t looked at the system yet, who knew nothing about it. If the organi-
zation is big enough, there are usually resources tasked with alternative
systems who can be brought in to check the work.
312 Core Software Security
It is worth mentioning again that one of the strongest tools security
architects have is peer review. It’s easy to miss something important. We
have instituted a system of peer review within multiple organizations at
which we have worked, such that any uncertainty in the analysis requires
a consensus of several experienced individuals. In this way, each assessor
can get his or her work checked and validated.
If there’s any uncertainty about any of the testing methodologies out-
lined here, getting an independent view may help validate the work or
find any holes in the approach.
9.5 Agile: Sprints
We believe that the key to producing secure software with an Agile process
is to integrate security into the Agile process from architecture through
testing. Rather than forcing Waterfall development on top of an Agile
process, security has to become Agile; security practitioners must let go
of rigid processes and enter into the dialog and collaboration that is the
essence of Agile development. Recognize that we have to trust and work
with Agile development teams, and make use of the Agile process rather
than fighting the process and its practitioners.
Figure 9.12 demonstrates how the archetypical SDL illustrated
in Figure 9.1 changes to reflect an Agile process, in this case, Scrum.
Requirements and architecture are a front-end process to Agile cycles, or
“Sprints.” Architecture feeds into the repeated Sprint cycles. At the end of
a series of Sprints, prerelease testing is applied. All the other tasks in the
SDL occur during each Sprint.
A Sprint is a cycle of development during which chunks of code—
“user stories”—are built. Each Scrum team chooses the periodicity of the
team’s Sprints. Typically, Sprints last somewhere between 2 and 6 weeks.
Each Sprint cycle is precisely the same length; this allows an implementa-
tion rhythm to develop. Whatever is not finished in its Sprint is put back
into the backlog of items waiting to be built. At the beginning of a Sprint,
some design work will take place; at least enough design needs to be in
place in order to begin coding. Still, the design may change as a Sprint
unfolds. Also, testing begins during the Sprint as soon as there is some-
thing to test. In this way, design, coding, and testing may all be occurring
in parallel. At the end of the Sprint, the team will examine the results of
the cycle in a process of continuous improvement.
Figure 9.12 Agile SDL.
314 Core Software Security
In Scrum, what is going to be built is considered during user story
creation. That’s the “early” part. A close relationship with the Product
Owner* is critical to get security user stories onto the backlog. This
relationship is also important during backlog prioritization. During the
Sprint planning meeting, the meeting at which items are pulled into the
development process (each “Sprint”) for build, a considerable amount of
the design is shaped. Make security an integral part of that process, either
through direct participation or by proxy.
Security experts need to make themselves available throughout a
Sprint to answer questions about implementation details, the correct
way to securely build user stories. Let designs emerge. Let them emerge
securely. As a respected member of the Scrum team, catching security
misses early will be appreciated, not resisted. The design and implementa-
tion is improved: The design will be more correct more often.
Part of the priority dialog must be the interplay between what is pos-
sible to build given the usual constraints of time and budget, and what
must be built in order to meet objectives, security and otherwise. The
security expert doesn’t enter in with the “One True Way,” but rather, with
an attitude of “How do we collectively get this all done, and done well
enough to satisfy the requirements?”
Finally, since writing secure code is very much a discipline and prac-
tice, appropriate testing and vulnerability assurance steps need to be a
part of every Sprint. We think that these need to be part of the definition
of “done.” A proposed definition of “done” might look something like
the following, based on the seven determining questions discussed earlier:
Definition of Done
1. All code has been manually reviewed (and defects fixed).
a. All code has been peer-reviewed.
b. Critical code has been reviewed by a senior or security subject-
matter expert.
* The Product Owner is a formal role in Scrum. This is the person who takes the cus-
tomer’s and user’s viewpoint. She or he is responsible for creating user stories and for
prioritization. A Product Owner might be an independent member of the develop-
ment team, a senior architect (though not typically) or a product manager, or similar.
It should not be someone who has hierarchical organizational power over develop-
ment team members.
..................Content has been hidden....................
You can't read the all page of ebook, please click here login for view all page.