As the product draws close to completion, an important question has to be answered: from a security and privacy perspective, is the product ready to ship to customers? The goal of the Final Security Review (FSR) is to answer this question. Performed by the central security team, the FSR is not only a critical part of the Security Development Lifecycle (SDL), it’s also complex, including many important tasks.
Before a software product can ship to customers, it must successfully complete its FSR. A failed FSR must be evaluated to determine how egregious the issues are. If they cannot be resolved by the product team, senior management must make the final call on resolving the issues.
The product group does not perform its own FSR. At Microsoft, the central security team performs the FSRs for all product groups.
The FSR is a review verifying that the product group has followed the SDL correctly during the product’s entire development lifecycle. The word “final” makes it sound as though this is where a good deal of security work is performed, but it really is not. Bug scrubbing should be done per bug as each one arises, threat-model reviews should be performed after implementation, and tools should be used as much as possible throughout the process by making them part of your build process. If the SDL is followed correctly, an FSR should be a short affair.
Important
For a software development team that has followed the SDL correctly, there should be few surprises during the FSR.
This short chapter outlines the major tasks required when performing a final security review. The following components make up the FSR process, and we’ll look at each in more detail.
This part of the FSR is not at all technical—it’s pure process, but it’s process that must be performed well for the FSR to run smoothly. First, the team must fill out a questionnaire. Examples of questions follow—note that many of the answers to these questions are known well before the FSR:
Is this product standalone or a service/management/feature/add-on pack?
Is any part of the product network-facing?
When was the security push, and how long did it take?
Where is the attack surface documented?
Where are the threat models located?
Where is the source code located?
Where is the bug bar documented?
Where is a list—preferably a database query—of unfixed security bugs?
Has the security team previously reviewed the threat models? If so, who did the review, and what was the outcome?
Are there any SDL requirements that you know the software is not compliant with? If so, what and why?
Did you run all analysis tools, and when was the last run of the tools?
This list is not exhaustive, but as you can see, the aim is to help the reviewers in the following FSR tasks.
Threat models—explained in detail in Chapter 9—are a cornerstone for building secure systems, and it is imperative that the models reflect reality. During the FSR, the threat models should again be reviewed to verify that they are accurate, up to date, and have appropriate mitigations in place. Refer to Table 9-12, to determine the threat-model relevance and quality.
Pay close attention to the last time the threat models were updated. A very old threat model is probably out of date and inaccurate. Also look at the data flow diagram. Does it look accurate and complete? If not, the entire threat model is inaccurate.
If developers make mistakes writing code, and authors make mistakes writing documentation, and testers make mistakes building tests, doesn’t it make sense that people will make mistakes when they enter bug information in the bug tracking database? The purpose of this part of the FSR is to verify that security bugs that are marked as “Won’t Fix” are appropriate to leave unfixed. All security bugs that exceed the specified threshold for the product must be fixed prior to releasing the product, so it’s important that the bug severity or criticality is correct. We have found that people make mistakes every now and then in entering a bug’s criticality, due to simple human error.
If you have a large number of bugs, you need to make sure that you have enough people reviewing the bugs in a timely manner. At Microsoft, we normally try to review all such bugs in less than a week, and we enroll enough security people to meet the allotted time. Don’t drag this phase out, even in the face of a large number of bugs.
Also note that you should mark each bug that is reviewed as such so that you don’t go back over the same bugs time after time. Assuming that you are using the bug-tracking fields defined in Chapter 6, you should create a new field named SecAudit with the following possible values:
Untriaged
Not a security concern
Defense in depth
Low severity
Medium severity
Important severity
Critical severity
Next, mark all unfixed security bugs (where Security/Privacy Bug Effect < > Not a Security Bug) with SecAudit = Untriaged.
As security people review each bug, they should set the SecAudit field to the appropriate value listed previously.
Once all the bugs have been reviewed, the results should be analyzed (triaged) to determine which bugs should be fixed. Invariably, there will be some unfixed bugs that are borderline; our only advice is to err on the side of fixing all security bugs that exceed the specified threshold for the product. Remember, you are protecting your customers!
It’s important for the FSR team to verify that all appropriate tools have been used during the development of the software. A list of the required tools and versions is given in Chapter 21. Use of appropriate security testing tools, such as file and network fuzzers, must also be reviewed and verified to ensure that they have been executed correctly.
There are many ways to do this: the first is just to look at makefiles and build scripts and verify that the appropriate flags are set. For example, in the case of the Microsoft Visual C++ tools, we make sure that the compiler command line includes /GS and no /GS- flag settings. We also check that the MIDL compiler uses /ROBUST and so on.
Some settings can—and should—be validated with tools, but some tools usage can be confirmed only by asking the development team. For example, static analysis tools require the development team to affirm that they used the tools.
You must document early in the process how you are going to determine compliance with tools requirements because this will vary if you use different tools.
Any compliance failure must be treated as a security bug and triaged accordingly. Assuming that a product team is trying to execute on the SDL responsibly, you should find no surprises during the FSR when validating tools usage.
At the end of the FSR, the security team will either agree that the FSR is completed and the product can be released to customers or determine that there are issues requiring remediation. If issues arise, each should be looked at to determine the correct course of action.
In some instances, a development team might reasonably be unable to comply with an SDL requirement and will ask for an exception. Often such exceptions are granted earlier than the FSR, but any exception should be looked at holistically; a single non-egregious issue—perhaps there are a small number of banned application programming interfaces (APIs) left in the code—is probably fine, but a multitude of issues or, perhaps, a banned API that is obviously vulnerable is not fine!
Handling exceptions is complex, and no easy-to-follow set of rules exists. Each exception should be decided on a case-by-case basis. However, an exception should never be granted without first conducting a thorough assessment of the impact on the product’s overall security if the exception is granted.
In general, if an exception is granted, the issue or issues should be remedied in the next public version of the product. This does not mean the next full version of the product—it means a service pack or a “dot release.”
When you’re deciding whether to allow an exception, it should not be uncommon to involve senior executives in the decision process. The pros and cons of each alternative resolution to the issue must be laid out so that upper management can make the call. You’ll need to gather all the appropriate background information to aid in making the correct decision, including:
What is the issue being resolved?
Why is the issue being found now? Was this unknown at a time when the issue could have been fixed?
What are the possible ramifications, if any, of not making the fix?
What should you tell customers about the issue? Do not rely on customers reading a readme file to stay secure.
What is the plan if the call is made to not resolve the issue, and the issue turns out to be worse than expected—or if an attack occurs?
What is the plan to fix the issue if it’s not resolved in the current product version?
When can customers obtain an updated version that fixes the issue?
Are there other mitigating factors that make the issue less important?
We can’t give concrete guidelines to any of these questions because there are none. Again, each exception decision must be made on a case-by-case basis.
Before a product ships, a pair of critical security eyes must validate that the software complies with the SDL security and privacy policies. An FSR can take a long time, but it’s critical that it be completed to make sure that the software has achieved all the SDL requirements for secure software. Remember that the FSR is not a penetration test, and it’s not the point at which you do a great deal of security work. It is a means to determine the overall ship quality and nothing more. At the end of the FSR, the product is either ready to ship, or there are issues that must be remedied. In some cases, exceptions might be granted for reasonable deviations from the SDL, but decisions to grant exceptions must not be made in haste and should be made only in light of all the possible ramifications of not fixing the deviation.