This chapter tells managers what they need to know about the Security Development Lifecycle (SDL). Our major focus is the role of managers in making the SDL succeed: what the manager or executive must do to ensure that his or her team can build more secure software.
Another purpose of this chapter is to prepare the manager or executive to deal with the impact of the SDL on development projects: what kinds of resources will be required, what impact the SDL will have on costs and schedules, and how the manager should assess whether the project is on track to comply with SDL requirements.
It is very important that managers understand the SDL’s impact on the software they produce and the expectations that SDL places on the stakeholders in their organizations. SDL is not free—it requires time, money, and the strong commitment of senior managers to prioritize security over other factors such as time to market and compatibility with older, less secure software versions. One key measure of success for this chapter is if a senior executive reads it and walks away saying: “Yes, that makes perfect business sense.”
In their “day jobs,” the authors often brief Microsoft customers and partners on the inner workings of the SDL, explaining how Microsoft is using the process to create more secure products. One question that we frequently hear is: “What component or aspect of the SDL has been most important to its success?” There is no easy answer to this because the SDL is an integrated process that comprises a large number of individual and equally important phases. The viability of the SDL within an organization such as Microsoft is intimately tied to the fact that each phase is necessary and contributes to improved product security. If any phase were unimportant, the SDL team would have to remove or modify it or face a loss of credibility for the entire process. Training, which is not so much a discrete phase as a cross-cutting component that affects all phases, usually makes the “extremely important” list, as does threat modeling.
Note
Every task within SDL is there for a reason: it leads to more secure software. If any task were deemed ineffective, it would be removed from SDL.
Still, executive commitment to the SDL process is the key factor for success. Bill Gates sent e-mail committing Microsoft to a sustained drive toward Trustworthy Computing, telling individual contributors, middle managers, and senior executives throughout the company that the rules and priorities for Microsoft’s development teams had changed. And the fact that the group and senior vice-presidents responsible for Microsoft Windows Server 2003 (and other products) decided to stop development and delay schedules in order to conduct security pushes and complete Final Security Reviews told everyone concerned that Microsoft was willing to do what was necessary—in terms of longer product schedules and extra staff effort—for improved security. Of course, shipping on time is extremely important at Microsoft (and in most other software companies), so this was a very significant change and one that managers might be expected to resist. The commitment of the most senior executives in the company to delaying products as needed to improve security told all concerned that Trustworthy Computing was a reality and not just a slogan. And the continued commitment by Microsoft’s top management to Trustworthy Computing and the SDL has led to a culture change at Microsoft—today, everyone knows that product groups must do what it takes to meet security requirements before they ship their products.
In the authors’ view, managers’ and executives’ most important contribution to the SDL is support for the process. And executive support for the SDL has been the key factor in making the process successful and effective.
To lead effectively at Microsoft, managers must understand and get involved in the issues and challenges that confront their organizations. This means that managers must understand the security problems that their products pose and commit to resolving those problems. The facts that motivated this commitment at Microsoft were discussed briefly in Chapter 3:
Customers were complaining about the frequency and cost of patching for security vulnerabilities.
Actual attacks manifested in the form of worms and viruses were disrupting customers’ IT operations to an extent that made the attacks very visible to both operations staffs and end users.
Press coverage was focusing on security problems to the point of overshadowing product improvements.
The need for frequent patching diverted Microsoft developers’ time and attention away from new features and new code and toward responding to security vulnerabilities. Although the costs of patch development were not great in absolute terms, the frequent need to divert developers to patching made development schedules less predictable—and conveyed to teams and their managers the dimensions of the security challenge.
In sum, real-world business considerations dictated that Microsoft address customers’ needs for improved security and more than justified the investment in the SDL. Not only is it more efficient to build watertight boats than to divert the shipwrights to plugging leaks, but the customers are much happier with the results.
Because implementing the SDL is expensive—and the authors have already made it pretty clear that it’s not cheap—any effective manager is going to ask whether the SDL is really necessary. The answer is, of course, “It depends.” If customers rely on the security of your software, then it behooves you to ensure that the software you supply is up to the challenge. Platform products—operating systems, database systems, e-mail and collaboration servers—obviously fall into this category because they must protect the confidentiality and integrity of user data and because the computing resource provided by the platform must remain available even in the face of hostile attack. But security is equally vital in other kinds of products, including e-commerce (Web) applications and many of the line-of-business applications that are used within organizations (where sensitive data must be handled and not all users are equally trusted or authorized). Many applications developed by government contractors handle sensitive information, which demands stringent security measures. Although the security of these applications could be circumvented if the underlying platform were insecure, attackers will target the applications themselves if security measures at the platform level make attacks there costly or infeasible.
The extent of the challenge that any particular software product or package faces depends on how exposed the software is to potential threats and on the value of the information that it is used to process. In Chapter 3, we summarized Microsoft’s tests for the applicability of the SDL to software products—software that is used in a business or organization, software that is exposed to the Internet, or software that is used to process sensitive or personal information. When considering the application of the SDL to in-house (line-of-business) applications, it’s important to focus on the impact on the business if the data were disclosed, modified, or destroyed. Recent attention in the media and by government and consumer advocates to disclosures of customer data have reemphasized the potential impact of security failures in e-commerce and line-of-business applications.
The security of Microsoft software has historically (at least since the mid-1990s) been exposed to special scrutiny because of its wide deployment in personal, business, and organizational environments. A researcher who discovers a vulnerability in Microsoft software might have found a way to attack millions of systems and their users! And the popularity of vulnerability research on Microsoft software has provided a degree of “cover” for smaller development organizations. However, recent (as of 2005 and 2006) trends in the discovery and publication of software vulnerabilities have demonstrated that if the work required to attack a Microsoft software product increases, the vulnerability seekers will look elsewhere.
Note
As Microsoft enhances the default security of its products, attackers are turning their attention to lower-hanging fruit—other software in which security best practices have not been followed.
In one case that the authors consider a significant success for the SDL, vulnerability finders radically reduced their attention to Microsoft SQL Server software—which had previously been a prime target—and focused their efforts on Oracle database products (Mogull 2006). Even if vulnerabilities remain in SQL Server—and we are sure that it has not yet achieved perfection—users of this particular Microsoft product are safer because new vulnerabilities are not being discovered or exploited. In a similar development, the SysAdmin, Audit, Networking, and Security (SANS) Institute reported that vulnerability finders were increasingly focusing on security and backup software that users had acquired with the expectation that such software would keep their systems and data safe (SANS 2005).
The fact that vulnerability finders will seek alternative targets whose vulnerabilities are easier to find than those in Microsoft software has a direct bearing on the priority of the SDL for both software vendors and enterprises’ internal IT development organizations. Vulnerability finders have never confined themselves exclusively to Microsoft software, and it’s clear that their attention to other vendors’ products is increasing: this trend is evident from the National Vulnerability Database maintained by the U. S. National Institute of Standards and Technology (NIST 2006). There have already been comments in the media to the effect that enterprise line-of-business IT systems are the next target for security researchers and security attacks. The developing trend toward targeted attacks focused on financial gain rather than on Internet vandalism also makes enterprise applications an appealing target.
You might want to know where it’s not necessary to apply the SDL in software development. The answer is that most software sold to customers or used by an organization or business should go through the process! Some standalone games or other entertainment products that don’t handle sensitive information or touch the Internet are exceptions, as are applications developed by an organization for very limited use (such as one-time data analysis applications). However, even software that does not otherwise qualify for application of the SDL should be reviewed to ensure that it does not somehow compromise or expose the platform on which it’s run—for example, by installing unprotected user accounts or modifiable executable files.
In summary, we believe that the rules that Microsoft uses (enterprise application; exposure to the Internet; sensitive information) to determine applicability of the SDL can serve as a guide for other organizations that are trying to decide whether they need to apply the SDL to their development efforts. Of course, we leave the precise determination of need to each product vendor or IT organization, but as a general principle, we believe that most software development organizations—whether vendors of packaged software, developers of e-commerce applications, or in-house line-of-business developers—will need to deploy a process similar to the SDL at some point.
If you have decided that it’s necessary to apply the SDL to some or all of the software that your organization develops, what do you as a manager have to do to ensure that your organization’s efforts are effective? The next few paragraphs summarize actions that we believe a manager should take to support the application of the SDL in his or her organization.
If you believe that it’s important for your development teams to implement the SDL and produce more secure software, you have to make that belief clear. At Microsoft, Bill Gates sent his Trustworthy Computing e-mail, but that was not all. Jim Allchin (then the group vice-president of the Platforms Group) personally kicked off the briefing/training session when we started to engage component team managers in the planning of the Windows security push, and Brian Valentine (the senior vice-president of the Windows Division) sent division-wide e-mails telling his employees what we were about and why it was important. The same story applies to the Microsoft Exchange and SQL Server products.
Note
Contrary to what you might read in the press, the various security pushes across Microsoft were not the method by which we shipped more secure software—they were simply the start of a long journey.
The preceding examples are specific to Microsoft, and although experience has shown that the methods introduced in this book are much more effective than conducting a one-shot security push, the need for an executive statement remains. The people who are designing, writing, and testing the code need to hear from you and understand that you believe the changes to their work involved in implementing the SDL are important, and why.
It is not sufficient to make a single statement about the importance of the SDL and expect your subordinates to follow through on their own. As with everything that managers do, follow-up is important. During the Microsoft Windows security push, Brian Valentine and the vice-presidents who worked for him sent status e-mails, held periodic meetings, and engaged with people across their organizations to remind them of how important security was to the success of their products. We identified the teams and individuals who were doing the best jobs of finding security bugs and recognized them with public statements and prizes. Today, with the SDL a normal part of product development at Microsoft, executives continue to communicate, in e-mails and team meetings, the importance of security and the necessity of meeting the requirements of the SDL. It’s very important for you to provide reminders, recognition, and rewards consistent with the culture of your organization to encourage effective execution of the SDL.
Again, our task of organizing and executing visible recognition during the many product security pushes was relatively simple. Security was the only thing the teams were working on, and the duration of each push was finite. But there are many milestones and deliverables in the SDL (threat models built, legacy code or interfaces removed or disabled, tests completed, bugs found) that offer opportunities for statements, recognition, and reinforcement by management. You should take advantage of these opportunities to ensure that your teams know what you expect and know that you are watching and intend to encourage behaviors that lead to more secure software.
Earlier, we mentioned that implementing the SDL had a significant impact on product schedules at Microsoft, and we’ll talk about how significant later. But you should know that there is a cost to implementing the elements of the SDL, and if you’re serious about shipping more secure software, you’ll need to pay that up-front cost. (We know that customer satisfaction with the security of Microsoft products that have undergone the SDL has improved, and we are confident that the SDL has more than paid for itself in improved customer satisfaction and reduced impact of vulnerabilities on customers and on Microsoft teams.) When you look at development schedules and tools budgets—to take two examples—elements of the SDL are almost certain to be visible if your teams are executing the process “right,” especially if they are just introducing the SDL to your development organizations. You’ll have the choice of signing off on those resources and making it clear that you believe they’re important, or “pushing back” and expecting your organization to implement the SDL on the cheap. Software security is an area in which you get what you pay for; if your teams are asking for the resources to implement the SDL in an honest and effective way, support them.
The most striking example of providing resources that we’ve seen came, again, during the Windows security push. As we were starting to plan the push, we held an initial meeting with one of the Windows Server 2003 project managers and got an offer to stop development for two working days while the teams working on the server product attended training, did code reviews, ran tools, built threat models, and did penetration testing. Given the magnitude of the product and that this was our initial security push, that offer was woefully inadequate to the need, and we persevered with a request for a longer security push. Eventually, we got a commitment for a four-week security push and began executing it. As the scope of the work became clear, it was evident that even a four-week push would not accomplish what was needed, and the senior management of the Windows Division (with the complete agreement of the project manager who’d initially made the offer of a two-day push) extended the duration of the push from four weeks to eight. Of course, we all had to do our homework regarding what kind of effort was needed, how long it would take, and why it was important—but the point was that management provided the resources that the work really demanded. The key aspect of this commitment is that the security push was finished when the work of the push was complete, not when some arbitrary date arrived.
We wish that this last component of effective commitment were never necessary, and in your organization it might not be. But one aspect of management support for the SDL is being prepared to recognize when a product is not ready to ship and making the decision to delay until it is. The suggestion to “stop products” might make you think of getting close to shipping and then realizing that the product isn’t secure enough, and that is indeed one possibility. But in reality, a decision to stop or delay shipping can come in a variety of flavors, at a variety of times, and for a variety of reasons.
The rates of discovery of security bugs might be unacceptably high, and you might need to allow more time for the rate to drop to the point at which your security review and penetration testing teams are being frustrated by the high level of security of the product they are reviewing and testing.
It might become evident that a feature not only is insufficiently secure, but that it will never be made sufficiently secure. You might have to make the hard decision to remove the feature from the product.
Your internal security team, or an outside security researcher, might discover a new class of vulnerability that neither you nor anyone else had previously known about. You might have to decide whether to ship with vulnerable code or to delay the product and eliminate the root cause of the problem.
Finally, a component or an entire product might fail the Final Security Review (FSR) that comes toward the end of the SDL and assesses “fitness to ship.” You might have to support the FSR team and delay the product until it can pass.
The best example we know of stopping a product—and one that took considerable commitment to security—was the case of the Microsoft .NET Framework and .NET common language runtime (CLR). Before there even was an SDL, and without knowing precisely what steps would address their security concerns, the management of the Framework team decided that the rate of discovery of security vulnerabilities in their code was too high and delayed their ship schedule, and they launched the first security push, at that time called a “security stand-down.” Not only were security bugs found and fixed, but the default configuration of the .NET Framework and CLR were made more conservative to maximize the chance of mitigating any defects that might be left in the code. (In the four years since its release—after the stand-down—the .NET Framework and CLR have been the subject of only three security updates addressing externally discovered vulnerabilities, and none of these was in the core framework runtime software.) We wish that the need to “stop ship” never arose, but we’ve lived through examples of each of the cases listed here and seen product group executives at Microsoft make the right decisions for security at the cost of schedule and/or features.
In this section, we provide you with some thoughts on the resources you’ll have to commit to implementing the SDL in your organization. Unfortunately, this section won’t give you prescriptive guidance about resource requirements for the SDL. The reason is that almost everything we can say about resources begins with “it depends . . .” We’ll also suggest how a manager or executive can tell whether a development project is on track in implementing the SDL.
We know from experience that there are a great many variables that determine the resources needed to implement the SDL, and we have a good idea of the ways in which they affect the resources needed to implement the SDL successfully. But we can’t yet give you a set of equations that says, for example: “For this code size and this implementation language and this level of exposure to the Internet, here’s the factor that you should apply to the planned development resource level to get to SDL compliance.” One reason is that we don’t really collect fine-grained resource data at Microsoft. Unlike defense contractors or other services firms that do development at an hourly rate, Microsoft typically assigns a team to own one or more products and allocates the cost of the team to the product. There’s no attempt to identify the finer-grained costs of individual activities within the development process. So the salaries of testers on Windows are charged to the current Windows release, whether those testers are testing new features for application compatibility or doing security “fuzz testing” as part of the SDL.
Despite our lack of quantitative guidance on the costs of the SDL, we can identify some of the factors that we know to make a difference in implementing the SDL effectively. We believe that most of these factors will be just common sense, but we hope that a few will surprise some readers:
Implementing the SDL is cheapest if it’s being applied to a project that is building a product or application from scratch. Your ability to choose languages, coding standards, and tools with security in mind—and to avoid mistakes rather than fix them—makes for efficiency in time and effort.
Similarly, applying the SDL to a mass of “legacy code” (written before security was a consideration) is expensive. The teams have to find problems, make changes, and then ensure that their changes don’t cause any problems, such as backward compatibility, with the correct functioning of the product or component.
From the two preceding items, it follows that the second and subsequent releases of code that has gone through the SDL should be less costly than the first. The first SDL release pays the price, in time and schedule, to clean up most of the latent problems, and subsequent releases can concentrate on preventing defects from being entered in the first place and on looking for newly discovered classes of vulnerabilities.
All things being equal, it will be easier to apply the SDL to a language that produces managed code—such as C#, Microsoft Visual Basic .NET, or Java—than to an unmanaged one such as C or C++. Although it’s absolutely not the case that managed code is guaranteed to be secure, there are classes of security vulnerabilities that developers can’t introduce in managed code. For example, it’s not possible to write a buffer overrun in C#, as it is in C or C++, although it is still possible for a developer to write an application in C# that is vulnerable to a SQL injection attack. The fact that it’s not possible to include certain kinds of vulnerabilities in an application written in managed code saves the effort that would otherwise be required to look for, find, and remove them.
It is often both better—that is, more secure—and cheaper to remove a feature than to fix it. Obviously, if you applied this rule in an extreme way, you would ship nothing at all, and it would be perfectly secure. That is not what we are suggesting. But if you are faced with a feature or component that presents significant security problems and is either not widely used or obsolete, it might be better to remove it or disable it by default than to bring it up to a level of security appropriate to the current threat environment. Note that disabling by default can be—but should not be—a crutch for shipping an insecure design or security bug–ridden code. If a feature or component is insecure and off by default, but you know that many users will enable it, you should either pay the price to fix it or bite the bullet and remove it.
Tools are almost always more effective and more efficient than a manual search for implementation vulnerabilities. This does not mean, however, that such tools are a panacea and find all security bugs. Tools can parse vast quantities of code rapidly, faster than a human could, but tools are no replacement for humans. You may have read or heard of the major Microsoft tools, such as PREfast, that we use to scan unmanaged code for buffer overruns and some other kinds of vulnerabilities. But we also build tools as needed when a new kind of vulnerability is discovered and we believe that it might occur throughout a product or set of products. For example, we’ve built testing tools to find and report classes of remote procedure call (RPC) vulnerabilities and scanning tools to find and report common errors in system configuration.
A product or component with a long history of security vulnerabilities, or a product that has exhibited security vulnerabilities resulting from design (rather than code) errors, is likely to be costly to bring to an acceptable level of security. It might be tempting to say that you have “done enough” to such a product, but this is the area in which you can easily deceive yourself. It’s much better to pay what it costs to find and remove vulnerabilities than to watch the new vulnerability reports keep rolling in!
Training helps to reduce the cost of security. This might sound obvious, but it’s still worth emphasizing: an effective training program will motivate your designers and developers to produce more secure software in the first place. The code reviews, tools, and testing will find fewer problems, and implementing the SDL will be quicker and cheaper.
Secure designs reduce the number of code-level errors that result in security vulnerabilities and reduce the severity of the vulnerabilities that remain. As a result, code reviews, tools, and testing will find fewer security vulnerabilities that need to be addressed.
As stated earlier, we don’t have solid guidance on the cost of implementing the SDL, but we do have a rough idea of what the SDL, as implemented on several products that have undergone the process, has cost Microsoft. Our best guess is that the SDL for a product that has significant legacy code and is going through the SDL for the first time might cost as much as 15 to 20 percent in schedule (and thus engineering effort). As you’d expect, the presence or absence of the factors outlined in the previous section can drive this resource level up or down.
We are confident that the “steady state” resource level required to implement the SDL on a product that was initially developed under the SDL, or has gone through one or two prior SDL releases, is significantly less than the 15 to 20 percent previously estimated. However, our experience with the SDL is not yet extensive enough for us to provide a confident estimate of how low the steady state cost will go. And we are also confident that the cost—in impact on customers, in impact on our reputation, and in lost sales—of not implementing the SDL would have significantly exceeded the costs of implementing it.
This section provides some suggestions for managing the execution of the SDL. How do you know whether the product team is executing the SDL effectively and building a more secure product?
Beginning with Chapter 5, and throughout Part II, we’ll discuss in detail the activities that make up the individual stages of the SDL. Each of the SDL’s stages requires the conduct of specific activities and the production of specific outputs, either in the form of documents (in a few cases) or of bugs in the project’s workflow system, that must be investigated and (in many cases) fixed. A manager or executive who is committed to applying the SDL to his or her products should pay close attention to those outputs to determine how the effort is actually going. The following list outlines some of the key measures that are helpful in assessing the quality of SDL implementation “on the way,” so that managers won’t be surprised when the product ships.
Track training attendance for your teams. If your training includes tests or qualifying exams, you should track scores also.
Track threat model production and quality early in development. You’ll have to have a specialized security team analogous to Microsoft’s Secure Windows Initiative team. One of their tasks should be to review threat models to ensure that they are not only present but effective in identifying potential vulnerabilities early in the process.
Monitor the rates and types of security bugs found during product design, development, and testing. Overall, is the number of real or potential security vulnerabilities dropping as the project reaches completion? Are there specific classes of vulnerabilities (either by type, such as buffer overrun or cross-site scripting, or by component) that are not dropping with the rest?
Track the impact of externally discovered vulnerabilities that affect your product. If there are earlier versions of your product in the field, or if there are similar products in the field, ask whether vulnerabilities similar to those discovered by outside vulnerability finders would have been present in your product if it had shipped under the current “plan of record.” Of course, you’ll find and fix such vulnerabilities before your product ships. But if your only clue to their presence was the “external find,” this suggests that your SDL process is not doing what it should.
Following these suggestions will give you a lot of data to review and should help you assess both the effectiveness of your process and the trend of your product toward readiness to ship. The keys to effective management and monitoring are to watch the numbers and trends and learn what they mean in terms of effective execution of the SDL—and the development of more secure software.
Executives and managers play a vital role in the implementation of the SDL in a software development organization. Management commitment is vital to a team’s success at implementing the SDL and producing more secure software. Measuring both costs and benefits of the SDL is difficult. Although there are not yet any authoritative guidelines about the impact of the SDL on project cost and schedule, monitoring the deliverables and activities associated with each phase in the SDL can give managers a clear idea whether the project is on track and how much the SDL is costing. Tracking external measures, such as customer satisfaction with security and the rate of security incidents affecting products and services, can give managers a similar understanding of the benefits of implementing the SDL.