Need for More Effective Trade Studies

Today's complex systems are multidisciplinary systems involving challenging missions, advanced technologies, significant uncertainties, and multiple stakeholders with conflicting objectives. Decision-making is central to generating creative alternatives, creating value, managing risks, and meeting affordability goals. Systems engineering trade-offs are needed throughout the system life cycle to inform these system decisions. In the absence of a formal framework, trade-off studies are sometimes performed in an ad hoc manner. Also, some systems engineers may not have an in-depth understanding of trade-off analysis techniques. As a result, some use unsound techniques.

This project began with a need identified by a professional society. The International Council on Systems Engineering (INCOSE) (www.incose.org) has nearly 10,000 members and about 95 members of its Corporate Advisory Board. The INCOSE Corporate Advisory Board documented the need for more effective trade studies. They believed there was a lack of best practices information that crossed the life cycle and aligned with ISO standard (ISO/IEC/IEEE 15288, 2015), the Systems Engineering Handbook (INCOSE, 2015), and the Systems Engineering Body of Knowledge (SEBok, Systems Engineering Body of Knowledge (SEBoK), 2015).

This textbook presents a Decision Management process based on decision theory and cost analysis best practices and is aligned with ISO/IEC 15288, the Systems Engineering Handbook, and the Systems Engineering Body of Knowledge. We introduce key concepts and demonstrate these trade-off analysis concepts in the different life cycle stages using illustrative examples from defense and commercial domains.

Audience

The audience for this book are graduate students (systems engineering, industrial engineering, engineering management, other engineering disciplines); professional systems engineers, operations analysts, project managers, and engineering managers; and undergraduate students (systems engineering, industrial engineering, engineering management, other engineering disciplines). We assume that the reader has had an introduction to systems engineering, an undergraduate knowledge in probability and statistics, a course in systems modeling, and a course in engineering economy and/or life cycle cost. However, Chapter 3 reviews probability and Chapter 4 presents important resource analysis techniques required for cost analysis and affordability analysis.

Themes

We had several major themes that provided the foundation for this book.

1. 1. Use standard SE terminology. We have attempted to use terminology from the ISO standard (ISO/IEC/IEEE 15288, 2015), the Systems Engineering Handbook (INCOSE, 2015), and the Systems Engineering Body of Knowledge (SEBok, Systems Engineering Body of Knowledge (SEBoK) wiki page, 2015).
2. 2. Avoid trade-off analysis mistakes of omission and commission. The mistakes of omission are errors made by not doing the right things. The mistakes of commission are errors made by doing the right things the wrong way.
3. 3. Use a decision management process. Systems decisions are made throughout the life cycle. Many of these systems decisions are difficult decisions that include multiple competing objectives, numerous stakeholders, substantial uncertainty, significant consequences, and high accountability. These decisions can benefit from a structured decision management process.
4. 4. Use decision analysis as the mathematical foundation for trade-off analyses. A credible trade-off analysis should be based on a sound mathematical foundation. Ad hoc methods and unsound mathematics provide a base of sand for a trade-off study and, therefore, a base of sand for the decision-makers. Since trade-off studies involve complex alternatives, multiple objectives, and major uncertainties, we believe that decision analysis is the operations research technique that provides this sound mathematical foundation for trade-off analyses.
5. 5. Explicitly define the decision opportunity. Every trade-off study begins with an implicit understanding of the problem or opportunity. The initial problem is never the final problem. It is important to clearly define the decision problem as a broader decision opportunity.
6. 6. Identify and structure decision objectives and measures. Once the opportunity is explicitly identified, the next step is to identify and structure the decision objectives of the decision-makers and stakeholders. The decision opportunity and stakeholder values determine the objectives. Measures that align with the objectives are required to perform the trade-off analysis.
7. 7. Develop creative, doable alternatives. The key to trade-off analysis is developing good alternatives that span the tradespace. The generation of the tradespace is a critical trade-off analysis task that requires participation of the entire trade-off analysis team and support from decision-makers, stakeholders, and subject matter experts.
8. 8. Include resource analysis in the trade-off analysis. Organizations do not have unlimited resources. Therefore, affordability analysis is almost always a critical part of the trade-off analysis.
9. 9. Explicitly consider uncertainty. Systems development, deployment, operation, and retirement involve many uncertainties. The systems life cycle may be years to decades. The major uncertainties include technology performance, integration with other systems, markets/missions, environments, and the actions of competitors/ adversaries.
10. 10. Identify the cost, value, schedule, and risk drivers. The purpose of a trade-off analysis is to provide insights for system decision-making. Decision-makers need to understand the cost, value, schedule, and risk drivers of the system.
11. 11. Provide an integrated for cost, value, and risk analysis. Unfortunately, most of the current systems engineering practice develops and performs separate cost, value, and risk analyses. We recommend an integrated framework for cost, value, and risk analysis.

Book Organization

The book is organized into three sections and a summary (Figure 1). The first section discusses the trade-off analysis foundations. Chapter 1 provides an introduction to trade-off analysis and includes common mistakes of commission and omission made in trade studies. Chapter 2 provides a conceptual framework for trade-off analysis and presents key decision theory concepts required for a sound mathematical foundation. As mentioned earlier, Chapter 3 reviews probability and Chapter 4 presents resource analysis techniques and affordability analysis.

The Decision Management process is presented in the second section of the book. Chapter 5 introduces the INCOSE Decision Management process and provides a detailed illustrative example of the process. Chapter 6 provides the principles and techniques for identifying the decision opportunity that the trade-off analysis supports. Chapter 7 provides principles and techniques for identifying objectives and value measures that assess how well the alternatives meet the objectives. These measures are the foundation for assessing the trade-offs. Chapter 8 reviews and evaluates the techniques for generating and evaluating alternatives. Many of these techniques are illustrated in the third section of the book. Chapter 9 illustrates a model for trade-off analysis that integrates value and cost analysis.

The third section provides trade-off analysis issues and illustrative examples in the life cycle stages. The scope and information available for trade-off analysis are different in each life cycle stage. Chapter 10 presents trade-off analysis methods to explore the trade-offs in the early life cycle stages when many system concepts and architectures need to be evaluated to determine the most affordable concept for further development. Chapter 11 presents processes and techniques for evaluating system architectures. Chapter 12 presents illustrative examples for system design trade-off analysis. Chapter 13 presents an illustrative sustainment model with deterministic and probabilistic analysis. Chapter 14 provides several illustrative examples of programmatic trade-offs that focus on system acceptance and termination.

Chapter 15 summarizes the major themes of the book and identifies some potential trends that may impact trade-off analyses in the future.

Course Outlines Using the Textbook

In this section, we offer some possible course outlines that could be developed using this textbook. Of course, the content presented in the course should be selected based on the academic/professional education program objectives and the course objectives. We present a notional set of course objectives and offer some potential course outlines.

Notional Course Objectives

1. Understand the role of trade-off analyses to support system decisions in each stage in the system life cycle.
2. Identify and avoid the mistakes of omission and commission in trade-off analysis.
3. Understand and use decision analysis as the mathematical foundation for trade-off analysis.
4. Understand the sources of uncertainty in the system life cycle and be able to identify, assess, and model uncertainty using probability.
5. Understand the advantages and disadvantages of common systems engineering approaches used to generate and evaluate system alternatives.
6. Identify and structure stakeholder objectives and develop single objective and multiobjective decision analysis models to evaluate alternatives.
7. Identify and define a system decision opportunity that requires a trade-off analysis.
8. Understand the advantages and disadvantages of tradespace exploration techniques for trade-off analysis of concepts, architectures, designs, operations, and retirement.
9. Understand the need for an integrated decision model that incorporates design features, value, cost, and risk.
10. Perform a trade-off analysis using the INCOSE Decision Management Process using both deterministic and probabilistic techniques.
11. Communicate the insights of a trade study and the important trade-offs to senior stakeholders and decision-makers.

Illustrative Academic Course Outlines

In addition to the course objectives, the coverage of course topics will depend on the role of the course in the curriculum (required or elective), the prerequisites, the location of the course (early or late in program), and the type of course (lecture, project, or combined). The textbook could be used to prepare for a capstone design course. The textbook presents more material that can probably be covered in a one semester course. I would recommend covering all of Chapters 1, 2, 5–7. The instructor would select the sections to read for other chapters. Depending on the academic curriculum, Chapters 3 and 4 could be reviewed or covered in more detail.

Illustrative Professional Short Course Outline

The textbook can also be used as a textbook/reference for professional short courses. The topics presented in the course would depend on the needs of the organization and the students' academic and professional backgrounds. The course could be taught as a seminar to present new material or as a project course with student's applying the material they learn in the course on a notional trade-off analysis or trade-off analyses they are working or will work in the future. The following outline is for a 1-week project course with trade-off analysis modeling using notional data (provided to students or developed by students).

Gregory S. Parnell, PhD, INCOSE Fellow
Editor
University of Arkansas
Fayetteville, AR
September 2016

Reference

1. ISO/IEC/IEEE 15288 (2015). Systems and Software Engineering – System Life Cycle Processes, International Organization for Standardization (ISO)/International Electrotechnical Commission (IEC)/Institute of Electrical and Electronics Engineers (IEEE), Geneva, Switzerland.
2. SEBoK (2015). Systems Engineering Body of Knowledge (SEBoK) wiki page. SEBoK: http://www.sebokwiki.org (accessed 06 June 2016).