15.2.1 Use Standard Systems Engineering Terminology

In addition to systems engineering, many engineering disciplines are involved in trade-off analyses. Therefore, it should not be surprising that many different terms are used. We have attempted to use terminology from the ISO standard (ISO/IEC/IEEE 15288, 2015), the Systems Engineering Handbook (INCOSE, 2015), and the Systems Engineering Body of Knowledge (SEBok, Systems Engineering Body of Knowledge (SEBoK) wiki page, 2015). We believe these are useful references, and we encourage our readers to use these to expand their systems engineering knowledge. Chapter authors have introduced new terms when they believed an important distinction or insight could be obtained with the new term. Of course, the key terms at the end of each chapter provide the definitions of these terms.

15.2.2 Avoid the Mistakes of Omission and Commission

There are several papers that identify the problems with trade-off studies. Using the INCOSE Decision Management Process, the most common trade-off study mistakes of omission and commission are identified and explained in Chapter 1 (Parnell et al., 2014). The mistakes of omission are errors made by not doing the right things. The mistakes of commission are errors made by doing the right things the wrong way. For each step in the decision process, Table 1.4 provides a list of the trade-off mistakes, the type of mistake (omission or commission), and the potential impacts.

These mistakes can have significant consequences in system design and management and, ultimately, on the program and the system. Unfortunately, it is quite common to find multiple mistakes made in trade-off studies with some errors leading to or cascading with other errors. These cascading mistakes can lead to adverse impacts for the trade-off study team, decision-makers, stakeholders, and, ultimately, the system or program. In addition, repeating these trade-off mistakes can undermine the credibility of the SE organization/enterprise.

15.2.3 Use a Decision Management Framework

As we described in Chapter 1, trade-off decisions are required throughout the system life cycle. Many systems engineering decisions are difficult decisions that include multiple competing objectives, numerous stakeholders, substantial uncertainty, significant consequences, and high accountability. These decisions can benefit from a structured decision management process. The purpose of a decision management process is “to provide a structured, analytical framework for objectively identifying, characterizing, and evaluating a set of alternatives for a decision at any point in the life cycle and selected the most beneficial course of action.” The decision management process uses the systems analysis process to perform the assessments (ISO/IEC/IEEE 15288, 2015). The process uses decision analysis to quantify the benefits and resource analysis to quantify the costs.

Chapter 5 introduces and illustrates the INCOSE Decision Management Process using a UAV example. The Decision Management Process provides the essential steps that should be considered for all major trade-off studies. Some of these steps may not be required in a particular study. We only illustrate one technique for each of the steps. However, we know there is no one way to perform trade-off analysis for all systems for all life cycle stages. Therefore, the rest of the book provides examples of trade-off analysis best practices in different life cycle stages using a variety of sound techniques. As the life cycle stages progress, the decision space decreases and there is an increase in the amount of data as design information, test, and operational data become available. Techniques used in later life cycle stages take advantage of this new information.

15.2.4 Use Decision Analysis as the Mathematical Foundation

We believe a credible trade-off analysis should be based on a sound mathematical foundation. Ad hoc methods and unsound mathematics provide an unsound foundation for providing insights for the decision-makers. Since trade-off studies involve complex alternatives, multiple objectives, and major uncertainties, we believe that decision analysis is the operations research technique that provides this sound mathematical foundation for trade-off analyses.

Decision analysis can quantify the value across the tradespace. Chapter 2 provides the foundations of decision analysis. The chapter reviews the mathematical assumptions of single and multiple values and utility. If all objectives can be converted to a single objective, usually dollars, then single-objective decision analysis is appropriate. If not, then multiple-objective decision analysis is the appropriate technique. However, we recommend the use of other analysis techniques to help explore the tradespace including design of experiments and optimization.

While decision analysis is the foundation for analyzing the tradespace, operational data, test data, and modeling and simulation data are critical for obtaining the scores on the value measures. This provides an important linkage between proposed systems assessed for trades and the analytical models and simulations that provide tradespace data for which real-world data does not yet exist, is not safe to collect, or is too expensive to obtain.

15.2.5 Explicitly Define the Decision Opportunity

Every trade-off study begins when an implicit understanding of the problem or opportunity. In our experience, the initial problem is never the final problem. Also, we believe that every problem can be viewed as a broader decision opportunity. Failure to identify the opportunity can result in rework of the study, the decision made without the benefit of the study, or even a lost opportunity for the organization or enterprise. Therefore, we recommend that the trade-off analysis team explicitly define the decision opportunity for each study and vet that definition with the decision-makers and key stakeholders.

Chapter 6 provides a summary of the best practices for opportunity definition. The chapter identifies the types of knowledge required for opportunity definition, the techniques for stakeholder analysis, and the tools to describe the opportunity from the systems engineering and decision analysis literature and practical applications.

15.2.6 Identify and Structure Decision Objectives and Measures

Once the opportunity is explicitly identified, the next step is to identify and structure the decision objectives of the decision-makers and stakeholders. The decision opportunity and our values determine the objectives. Identifying the objectives can be especially challenging if the opportunity is new to the organization or enterprise. Generally, the more complex and important the opportunity, the more decision-makers and stakeholders are faced with conflicting objectives. Structuring the objectives provides a visual framework that helps decision-makers and stakeholders understand and validate the completeness of the objectives.

Chapter 7 describes the best practices for defining and structuring objectives and then developing value measures that measure attainment of the objectives. Defining the value of the system's products and services is one of the most critical tasks of systems engineering. In systems engineering, the objectives include the business/mission objectives, the stakeholder objectives, and the system objectives. The business/mission objectives are derived from the system purpose for the organization and its customers (strategic level). The stakeholder objectives include the goals of the other important stakeholders in the system life cycle (operational level). Finally, the system objectives include the technical objectives that are necessary for a system to meet the business/mission and stakeholder objectives in the system life cycle (tactical level).

15.2.7 Identify Creative, Doable Alternatives

The key to trade-analysis is developing good alternatives that span the tradespace. A trade-off analysis should not be an advocacy for a predetermined alternative. If we analyze poor alternatives, we may only select the best of a bad set of alternatives. If we only consider the baseline or the baseline and a couple ad hoc alternatives, we may miss the opportunity to create value. The development of alternatives is a critical trade-off analysis task that requires participation of the entire trade-off analysis team and support from decision-makers, stakeholders, and subject matter experts.

Chapter 8 provides the best practice for alternative development. The best practice to develop alternatives is a two-phase process. The first phase is the creative or divergent phase. The purpose of this phase is to identify potential ideas that would create great value. The second phase is the analytical or convergent phase. The purpose of this phase is to develop doable alternatives. The chapter also identifies the techniques that provide explicit procedures to develop creative, doable alternatives.

15.2.8 Use the Most Appropriate Modeling and Simulation Technique for the Life Cycle Stage

In early life cycle stages, the alternatives are very different and the data available on new concepts is limited. As we proceed through the life cycle, the range of alternatives narrows and the data available on the alternatives increase as a concept is selected, an architecture is defined, a design is performed, development test data becomes available, systems are produced, systems are deployed, and operational data becomes available. As a result, different modeling and simulation techniques may be more appropriate in different life cycle stages. The trade-off analysis techniques used in the illustrative examples in Chapters 10–14 illustrated modeling and simulation techniques the authors have found to be the most useful in each stage.

15.2.9 Include Resource Analysis in the Trade-Off Analysis

Organizations do not have unlimited resources. Therefore, resource analysis is almost always a part of the trade-off analysis. Chapter 4 presents the important resource analysis techniques and also presents the key concepts of affordability analysis. We believe that all systems engineers should understand the cost analysis techniques in this chapter. However, for large trade-off analyses, we recommend that one or more cost analysts be assigned to the team.

This book provides numerous examples of comparing the cost versus the value of the alternatives. In most cases, the cost will be the life cycle cost. This plot provides essential information that senior decision-makers can use to determine the affordability of the alternatives.

15.2.10 Explicitly Consider Uncertainty

Systems development, deployment, operation, and retirement involve many uncertainties. The systems life cycle may be years to decades. For longer life cycles, we should expect more uncertainty. The major uncertainties include technology performance, integration with other systems, markets/missions, environments, and the actions of competitors/adversaries. Surprisingly, many trade-off analyses use only deterministic methods and do not explicitly consider uncertainty. According to our literature review and reinforced by our experience, cost analysis makes the most use of uncertainty analysis. Monte Carlo simulation is the most used technique.

Uncertainty analysis is a major topic of this book. Chapter 3 provides an introduction to uncertainty modeling for systems decisions. Several of the chapters address uncertainty models including Chapters 4, 5, and 8–14. Chapters 5, 13, and 14 provide uncertainty analysis with Monte Carlo simulation using an Excel add-in. Chapter 9 provides uncertainty analysis using Probability Management.

15.2.11 Identify the Cost, Value, Schedule, and Risk Drivers

The purpose of a trade-off analysis is to provide insights to aid in system decision-making. Decision-makers need to understand the cost, value, schedule, and risk drivers of the system. When properly focused on the decision objectives, trade-off analyses can identify these insights. These insights are critical to cost, value, schedule, and risk management.

Many of the chapters in this book identify, describe, and illustrate techniques to identify cost, value, and risk drivers. When combined with schedule models, these techniques can also be used to identify schedule drivers.

15.2.12 Provide an Integrated Framework for Cost, Value, and Risk Analyses

Unfortunately, most of the current systems engineering practice develops and performs separate cost, value, and risk analyses. The cost and value models are usually quantitative models. Many times, the risk analysis framework is only a likelihood consequence matrix. This practice may help summarize insights to decision-makers; however, it does not identify the critical dependences. Therefore, it does not provide the critical information that decision-makers need to understand the impacts of the cost, value, risk, and schedule drivers.

We recommend an integrated framework for cost, value, and risk analysis. In Chapter 9, we illustrate an integrated framework where the same decision factors and uncertainties are simultaneously propagated through value and cost models. We believe that this method provides the most promise to provide credible, traceable insights to decision-makers and stakeholders.

15.3.1 Education and Training of Systems Engineers

This book is dedicated to providing a resource to improve the education and training of systems engineers and systems analysts who need to perform trade-off analyses. It provides the basic descriptions of methods, techniques, and tools as well as examples of practical applications using illustrative examples. The book can be used in numerous ways to further the education of the students and training of practitioners. Examples can be found in the “Trade-off Analysis Course Outlines” section of the preface. We have attempted to provide the necessary mathematical background for most of the techniques: for example, decision analysis (Chapter 2), probability and Monte Carlo analysis (Chapter 3), and resource analysis (Chapter 4). However, we have assumed an understanding of optimization (Chapters 10–13 and design of experiments (Chapter 12).

Education provides systems engineers with the understanding of basic and cutting-edge systems engineering methods, techniques, and tools. However, it takes exposure to real-world applications to develop and solidify one's skills as a systems engineer. Capstone projects in education programs, summer coops, as well as on-the-job training and mentoring are some of the best environments for honing one's systems engineering skills. Many organizations have professional continuing education and training programs to help increase the skill sets of their systems engineers. This book could be used to develop one or more modules for use in these programs. Figure 15.1 is an example layout for an Army organization's training program used to guide the development of systems engineers. It is an adaptation of a program used to train operations research analysts at the Center for Army Analysis.

15.3.2 Systems Engineering Methodologies and Tools

Modeling and simulation have been discussed as a method for analysts to test theories and explore concepts and tradespaces. Other methods used by systems engineers in various SE processes include functions-based systems engineering, object-oriented systems engineering, prototyping, interface management, integrated product and process development, lean systems engineering, agile systems engineering, and model-based systems engineering (MBSE) (INCOSE, INCOSE Systems Engineering Handbook, 2015, pp. 180–210). MBSE is a method that has been gaining in popularity since the publication of Wymore's book in 1993 entitled, Model-Based Systems Engineering.

MBSE is defined in the INCOSE Systems Engineering Vision 2025 as “the formalized application of modeling to support system requirements, design, analysis, verification, and validation activities beginning in the conceptual phase and continuing throughout development and later life cycle phases” (INCOSE, Systems Engineering Vision 2020, 2007, p. 15). Although the implementation of systems engineering practices varies by organization and system type, the use of model-based engineering has helped with cross-fertilization and standardization of methods, techniques, and tools. This has occurred as collaboration between engineers increases as they leverage information and data generated from domain models to feed models of other subsystems or systems. To accomplish this, engineers link models and the interfaces between models and data sets to help simulate the complexities of larger systems.

This linkage requires integration across organizations, disciplines, and phases of system development. Although this helps to provide insights into how a system might integrate with other systems or perform in the real world, it creates additional challenges for systems engineers (INCOSE, SE Vision 2025, 2014, p. 22). Systems engineers are often used to perform the human and model integration. When done effectively, the integration of data, algorithms, and process results in a tradespace visualization that enables insights into viable alternatives for decision-makers.

As systems engineers must continue to leverage modeling and simulation to generate more complex and viable tradespaces, the term MBSE will likely become synonymous with systems engineering. Although it will be only one of the methods used by systems engineers, its broad application will make it part of the standard tool kit along with stakeholder and statistical analysis.

To be clear, MBSE is neither new nor is it the panacea of systems engineering. Long discusses four basic myths of MBSE: (i) models are new; (ii) documents are disappearing; (iii) there is only one kind of model; and (iv) Systems Modeling Language (SysML) equals MBSE (Long, 2015). This book has illustrated numerous types of models that have been used for decades (e.g., cost models and value models) and enhanced or new models (e.g., probability management and tradespace exploration) developed to meet the emergent needs of analysts to answer the questions of decision-makers. Although static documents capture a point in time and are difficult to maintain when changes are required, it is unlikely that they will disappear anytime soon. It is more likely that they will be replaced by dynamic documents generated as required from a digital thread of information as required. This will facilitate one's ability to understand the process and decisions made during the life cycle of a system while facilitating the development or enhancement of evolving systems. The use of SysML, or a similar system description language, to facilitate discussions between engineers and the interface between models is essential; however, it is merely a component of MBSE. MBSE requires systems descriptions to help to identify the linkage between models, network(s), data sets, analytical tools, and data/information visualization tools in order to generate the data thread used to create viable tradespaces and information. Furthermore, the MBSE requires a repeatable process to ensure its soundness and validity as a systems engineer methodology.

New methodologies often require new or enhanced analytical tools to help implement. Tradespace tools reside in this paradigm. Numerous tools are under development by academia, industry, and government to help generate, analysis, and visualize tradespace data. Some of the statistical or tradespace tools developed by industry and/or academia include the following: R, Analytics, CyDesign Studio, DecisionTools Suite 6.0, Excel®, SAS®, JMP®, Relational-Oriented Systems Engineering and Technology Trade-Off Analysis (ROSETTA), and Risk Solver Pro. Government sponsored tools include the Advanced Systems Engineering Capability (ASEC), ARL Trade Space Visualizer (ATSV), Capability Portfolio Analysis Tool (CPAT), Engineered Resilient Systems (ERS) Tradespace Tool, Framework for Assessing Cost and Technology (FACT), Tradespace Analysis for Capabilities, Effectiveness, and Resources (TRACER), and Whole System Trades Analysis Tool (WSTAT).

The DoD-sponsored ERS Tradespace Tools package is an example of a cooperative effort between industry, academia, and DoD. It is dedicated to developing decision support methods as well as a collaborative MBSE Tradespace toolset based on an architecture open to organizations that work in partnership on DoD acquisition efforts. (Ender et al., 2015, p. 7)

15.3.3 Emergent Tradespace Factors

The increasing complexity, time to develop new systems, and cost of systems often require senior leaders to seek additional insights about systems under development. Previously, many of these insights were virtually impossible or too time-consuming to assess or not relevant to leader's decision-making process. However, as funding becomes more constrained and systems are used for extended periods of time or expected to operate in numerous environments, systems engineers are asked to expand the tradespace to include factors such as resilience and flexibility.

Depending on the community of interest, resilience can be defined in numerous ways. For example, fixed facilities or systems may define resilience as the ability to withstand or quickly recover from a natural disaster. For a nonstatic system, such as a transportation or DoD weapon(s) platform, “a resilience system is trusted and effective out of the box, can be used in a wide range of contexts, is easily adapted to many others through reconfiguration and/or replacement, and has a graceful and detectable degradation of function” (Goerger et al., 2014, p. 871).

Research is underway by industry, academia, and the federal government to help develop standard definitions within communities of interest for metrics such as resilience. This is important as analysts seek valid data to assess measures for system objectives. Industry is interested in these efforts as it seeks to understand the needs of its clients. This is true for its commercial as well as its government clients. Thus, industry is working with academia and government researchers on these efforts. An example of this is ERS.

Other system characteristics will be added to the tradespace in the future depending on the business model or purpose of the organization. Possible metrics include flexibility, manufacturability, deployability, sustainability, and easy of modification. Many of these metrics are still being defined and validated for their impact on system viability. Research also seeks to better understand the interrelationships between the metrics to help generate more valid tradespaces. For instance, the ease of building a system using additive manufacturing techniques may have a positive correlation to the maintainability and sustainability of the system. However, the initial cost of each copy of the system may be higher than systems built using more traditional methods. Conversely, the long-term cost to maintain the less expensive to build system could exponentially be more expensive than the system built using additive manufacturing. Of course, the exact opposite relationship may exist. Further research will help analysts better understand these new relationships and use them to help inform decision-makers of the most viable trades.