10.1.1 Key Concepts, Concept Trade-Offs, and Concept Exploration

Systems engineering trade-offs tend to be poorly defined and not well understood (Chapter 1). Not surprisingly, they are underemphasized in technical project management and often become a significant source of conflict among stakeholders as the project progresses. The inevitable outcome of ignoring trade-off analysis is selection of a concept that has low probability of meeting the needs and has high probability of cost overruns that sometimes lead to outright program failures.

At its core, trade-off analysis (or tradespace analysis as it is often referred to in systems engineering) is an analytical method for evaluating and comparing system designs based on stakeholder-defined criteria. Trade-off analysis requires measurable system attributes (also called measures of performance (MOPs) and measures of effectiveness (MOEs)). An example of MOP is “speed in engineering units,” while an example of MOE is “probability of mission accomplishments” in terms that directly reflect stakeholder value. These are quantifiable criteria that are useful in characterizing how a stakeholder values important system attributes (e.g., performance, cost, and ilities). “Ilities” are quality attributes or properties of an engineered system that contribute to system value once the system is put into use. Examples of ilities are resilience, interoperability, and flexibility. Ilities are nonfunctional system requirements that are concerned with broader system impacts with respect to time and stakeholders. While system attributes and decision-makers' utility functions capture the system need to a large extent, there are other sources of unarticulated or poorly characterized value. These tend to be quality attributes such as flexibility, extensibility, and resilience. Attributes such as flexibility and resilience are not easily defined, quantified, or costed. Flexibility offers the ability to respond to expected change. Adaptability provides the ability to respond to unexpected change. Resilience is the ability to respond to disruptions (both expected and unexpected). Therefore, flexibility and adaptability are enablers of resilience. In this regard, tradespace exploration techniques are likely to provide insights by affording the system designer a larger system view that includes relationships between architectural and design choices. The tradespace shifts as a result of changes in stakeholder preferences. This shift can potentially produce a change in the ranking of candidate design options. Comparing the relative shift in the ranking of candidate designs along an isocost can provide insights into how a candidate design may or may not be susceptible to such changes.

Figure 10.3 is a depiction of the tradespace for a Space-Based Radar (Spaulding, 2003). This tradespace is analyzed to determine the best design (Roberts, 2003; Shah, 2004). Typical analysis methods choose from among candidate designs that lie on the Pareto Frontier. The choice is invariably based on the option that has the lowest cost. However, when we consider other attributes, this may not always be the case.

With respect to Figure 10.3, a closer examination tells a somewhat different story if flexibility is deemed an important consideration. Flexibility implies that a system can be readily transformed into a different but related system at a modest cost. If flexibility is desirable, then Pareto Frontier solutions may not be the best. Roberts (2003) and Shah (2004) argue that these points in a sense are optimized for best value at a point in time and do not account for changes that might occur in the future. The latter, in fact, is the key motivation for wanting flexibility. While it may be appealing to incorporate flexibility into the utility model, one can potentially run into mathematical challenges (McManus et al., 2007). Therefore, treating quality attributes as separate criteria to be considered in addition to benefit and cost may provide more decision insights. Of course, all quality attributes including flexibility tend to come at a cost (Neches & Madni, 2013). So, the question that needs to be answered is whether or not the solutions on the Pareto Frontier are sufficiently flexible and, if so, what does flexibility cost. With respect to Figure 10.3, candidate designs A, B, C, D, and E all lie on the Pareto Front. However, close examination reveals that they are somewhat rigid in that they are not easily transitioned to other solutions on the Pareto Front. However, candidate option C, which has the same utility as option A but costs more than option A, can be “transitioned” to solution points B, D, or E more easily than option A. In this case, the cost differential between option C (the dominated point) and option A can be considered the cost of flexibility to move up the Pareto frontier. The generalized qualification of path dependence in transitioning between candidate solution options in a tradespace is an area of ongoing research (Ross, 2006; Ross & Hastings, 2006; Fitzgerald et al., 2012). It is intended to explicate the costs and benefits of including system quality attributes such as flexibility into an architecture or design. The rigid designs are those that have fewer design options to transition to than flexible designs do.

In the context of trade-off studies, measurable system attributes are used to determine how to balance stakeholder preferences (e.g., functions, objectives, measures, and requirements) to identify the system design that most closely satisfies stakeholders' objectives.

With respect to systems engineering trade-off analysis, a trade-off hyperspace is created to address the multiple trade-off factors (i.e., system attributes) for a system. Figure 10.4 presents a partial trade-off hyperspace for an unmanned vehicle. In other words, the trade-off hyperspace (i.e., tradespace) shown in this figure is meant to be a representative, not exhaustive. Different potential unmanned vehicles will display different trade-offs among these and other factors. Design features are represented in the design tradespace, while performance factors such as system performance, mission effectiveness, cost, schedule, and risk are represented in the performance tradespace.

As can be expected, systems engineering trade-off analysis is complicated by several factors. These include the following: multiple stakeholders with conflicting objectives and differing preferences for system functions and attribute level; uncertainty surrounding technology maturity and the future actions of adversaries/competitors, interactions among hardware, software, and human elements in the system; and the need to address trade-offs at multiple levels of abstraction and spanning multiple domains (conceptual, physical, social). It is important to note that Trade-off analysis in the concept space needs to be employed in several systems engineering contexts:

• When creating an unprecedented system
• When integrating legacy components into a new system
• When developing and establishing feasibility of system design and CONOPS

Trade-off analysis is further complicated by the number of system attributes in the tradespace, number of stakeholders, percentage of nonengineer stakeholders, presence of uncertainty in key attributes and the future environment, and whether or not the trade-offs vary with context. The concept space is typically explored in the conceptual design stage (also called upfront engineering). During conceptual design, the mapping from function to form occurs. As noted earlier, the physical form (i.e., implementation) determines most of the performance potential, cost, and schedule of subsequent development in the life cycle. A suboptimal decision in conceptual design can potentially result in cost and schedule overruns, as changes become increasingly more difficult to make later in the system development life cycle. The design concept and high-level specification are the outputs of this stage. These outputs serve as inputs to the preliminary design stage. The design trade-offs space (or tradespace, for short) is the space that is spanned by the fully enumerated set of design variables. Given a set of design variables, the tradespace is the space of possible design options. It is from this space that the initial system design concept is created/selected. The system design concept that is selected from the conceptual design tradespace is one that mitigates the likelihood of costly change later in the system life cycle and that maximizes stakeholder value. In this stage, intentional or inadvertent premature reduction of the design space can result in loss of potential future value (Chapter 1). Such premature pruning of the design options tradespace can preclude the realization of high value, risk-mitigated system.

A fundamentally important activity is the linking of the concept trade-offs to specific systems engineering life cycle phases and appropriate stakeholders. These links ensure that the right factors and considerations are addressed in the early stages of systems engineering, that is, conceptual design or preliminary design. These links serve to inform stakeholders about the applicable trade-offs that address variables of interest to them. This timely communication of trade-offs and subsequent decisions are essential for stakeholder understanding and, ultimately, stakeholder buy-in.

The creation of this link can be aided by defining a system ontology that spans system life cycle phases, tradespace attributes/variables, stakeholders and their interests in specific attributes, the weights assigned to the various attributes, and the supporting rationale.

Concept trade-offs apply to both system design and operation. Concept trade-offs during operation are often dynamic and can vary with changes in context (Madni & Madni, 2004). In the spirit of true concurrent design, system CONOPS need to be defined at the same time that the system is specified. This is key to assessing the feasibility of the design and CONOPS that result from a systematized analysis of the tradespace and sensitivity analysis of key design variables and their impact on key MOPs and MOEs. It is important to realize that at this stage, there is a high degree of uncertainty and ambiguity that is gradually reduced through modeling and simulation-based analysis. At this stage, great care needs to be exercised to resist premature elimination of options through untimely imposition of constraints during conceptual trade-offs analysis.

The outcome of conceptual trade-off analysis is an assessment in terms of feasibility, short-term impact, and long-term viability including affordability (Chapter 4). A variety of trade-off analysis techniques can be employed as the system moves through a series of architectural abstractions, that is, concept, function, form, and fit. Finally, it is imperative to go beyond the obvious technical and programmatic trade-offs to include the enterprise, social, and human cognitive dimensions and their impact on technical and programmatic trade-offs.

10.3.1 Storytelling-Enabled Tradespace Exploration

Storytelling is a powerful instrument for uncovering variables that should become part of the tradespace and for performing dynamic trade-offs (Madni, 2015a). Storytelling brings the benefit of illuminating changing contexts over time. Stories are created from a combination of mission description, use cases, and domain expert knowledge elicitation (Madni et al., 2014a). Interactive stories, that is, stories that allow users to interact with the system and thereby shape story evolution, are especially useful in uncovering previously unknown interactions as well as “hotspots.” Stories allow simplification (e.g., through abstraction and selective timeline compression) and thereby allow the user to uncover interesting interactions and “surprises” (both good and bad outcomes) with fewer runs than possible with traditional simulation (Madni et al., 2014b). The challenge is to ensure that the story is rich enough (i.e., complicated enough) to be interesting and insightful and simple enough to find short-term answers of interest to the different stakeholders. Because of these properties, stories add temporal, spatial, and human dimensions to the tradespace.

10.3.2 Decisions and Outcomes

Ward Edwards said that “a good decision cannot guarantee a good outcome” (Edwards, 1962). Real decisions invariably involve uncertainty. A decision is, therefore, a bet, and evaluating whether it is good or bad depends on the stakes and the odds, not on the outcome. Also, the goodness of a decision is with respect to the criteria one defines. The possible results from the resolution of an uncertain event are called outcomes. With some uncertain events, there are only a few possible outcomes. In other cases, the outcome is a value within a specified range. The outcome of an uncertain event comes from a range of possible values and may fall anywhere within that range. Multiple uncertain events can potentially be considered in a decision situation. However, only a subset might be relevant. For an uncertain event to be deemed relevant, it must have some impact on at least one of the objectives. An example of an uncertain event in systems engineering is the time for a technology to mature from TRL3 to TRL7. This uncertain event has an impact on system objectives such as system performance, system affordability, system scalability, and possibly other system quality attributes (also called ilities). The complexity of a decision situation depends on the number of relevant uncertain events. The larger the number of uncertain relevant events in a particular context, the more complex the decision. Early in the system life cycle, such as conceptual design, the number of uncertain events is at its largest; some of these events will resolve themselves before the system enters operations (e.g., system test results, while others will resolve during operations (e.g., adversary/competitor actions). Also, some relevant uncertain events may depend on other such events. This dependence further adds to the complexity of the decision. Decisions made during the concept stage then are not only far-reaching, but also made complex by the sheer number and variety of uncertain events.

10.3.3 Contingent Decision-Making

In systems engineering, proactively managing the impact of uncertainty is critical. This capability is especially crucial when exploring the concept space and performing conceptual trade-offs analysis. One method of managing uncertainty is to create flexible opportunities that allow for deferring some decisions until uncertainties are resolved. Exploring the concept space and performing conceptual trade-offs analysis provide mechanisms to identify opportunities for deferring decisions and to define what have been described as “real options” (Myers, 1977)

Figure 10.6 illustrates some of the key concepts of a real option, including implementation of the option (real option enabled), the period during which the option enables the “right but not obligation” to execute a decision (at some cost), followed by the end of the option (real option expires).

In systems engineering, real options have been exploited to hedge against downside risk while promoting upside opportunity. However, such flexibility, similarly to all quality attributes, comes at a cost. The cost in this case is increased system complexity, additional financial outlay, and potential delay in schedule. The challenge is where to embed the real options to derive maximum leverage and enable the ability to satisfy the appropriate objectives at the appropriate time. Maximum leverage is derived by having key options available when they are needed (often as late as possible in the system life cycle, when cost of change is at its highest). During the option-enabled period, there may be associated carrying costs (e.g., storage costs for spare satellites, which enable the option to launch replacements to an existing constellation). Having such options allows for contingent decision-making where decisions can be deferred and executed contingent on emergent information. The challenge for designers is determining which options to embed in the system in order to respond to both anticipated and unanticipated uncertainties (Ricci et al., 2013).

10.7.1 Tradespace Exploration and Sensitivity Analysis

Tradespaces can be represented using a variety of visualization formats, including Kiviat Charts, bar charts, and utility–cost scatterplots. The utility–cost plot representation of the tradespace is a concise representation that highlights two critical decision metrics. The value-oriented focus of the tradespace plot is that utility, a key decision metric, is defined by the decision-maker who ultimately chooses from among the available alternatives. It is important to recognize that with designer-specified metrics such as speed or power; utility has to be inferred because these engineering characteristics used by designers do not directly measure the value or benefit that accrues to owners of the system when it is operated to achieve mission objectives.

The value-centric representation of a tradespace allows for the comparison of different concepts on the same basis (e.g., utility and cost). Figure 10.8 illustrates a tradespace for an operationally responsive disaster surveillance system, with different concepts dominating different regions for the Owner stakeholder. In this case, the satellite concept delivers less benefit and more cost than many of the alternatives. Surprisingly, the sensor swarm concept appears most promising at low cost and high benefit (however, further inspection reveals that the swarm is difficult to deploy in various potential mission areas, making the concept infeasible. This example illustrates the importance of iterative exploration of tradespaces, especially when utilizing low fidelity models for evaluating costs and benefits. Assumptions made during concept development may not be compatible with new information discovered later in the life cycle; therefore, it is essential to explore the uncertainties and sensitivities in tradespace results.

When a requirement or a stakeholder's preference changes, the impact of that change on the tradespace can be rapidly assessed. Ross & Hastings (2005) show a representative tradespace impact of a change in the utility function for a system after the decision-maker reviewed the initially proposed design (Figure 10.9). Another use of the tradespace plot is to compare point designs in a macro-tradespace sense, that is, in terms of high-level alternatives. Designs that are dominated (i.e., inferior to the Pareto front) can be readily identified and examined.

10.7.2 Tradespace Exploration and Uncertainty

Uncertainty complicates tradespace exploration and alternative selection (Chapter 9). Various approaches have been pursued to include uncertainty in tradespace representation and exploration. Walton (2002), Walton & Hastings (2004) employs Monte Carlo simulations to generate probability distributions associated with each point in the tradespace. In this case, uncertainty is quantified as the variance of the value metric. Once uncertainty is quantified, system designers can exploit both the upside and downside of uncertainty (Walton & Hastings, 2001). For example, portfolio of system designs that combine options with anticorrelated uncertainties will have lower uncertainty than the uncertainty in individual designs (Walton, 2002; Walton & Hastings, 2004). Figure 10.10 presents an example of uncertainty-value space from which portfolios are constructed.

Each point in the figure is a point in the tradespace. As with financial portfolio theory, the efficient frontier of maximum value for a given level of uncertainty represents various combinations (or portfolios) of options. For financial instruments such as stocks and bonds, the uncertainty, or variance, is equated with “risk” and the expected value is called “return.” Portfolios with best risk–return ratios are constructed based on the risk tolerance of the portfolio holder (Howard & Abbas, 2015, pp. 622–624). In Figure 10.10, three types of decision-makers are shown: highly risk-averse, moderately risk-averse, and minimally risk-averse. The optimal investment strategy for each type of decision-maker is determined by the intersection of the isovalue contours of the decision-maker with the efficient frontier. The analogy is made between partial ownership of financial instruments (i.e., stocks, bonds) and the partial investment in the development of physical systems. As uncertainties are resolved with further development of the system, the portfolio is modified accordingly (Walton, 2002). The key idea behind the uncertainty-oriented view of the tradespace is to develop an understanding of high leverage points for technology insertion or investment. Technology development that results in more anticorrelated design options can potentially mitigate uncertainty and help system engineers and program managers protect against program failure that is more likely with correlated technology development.

One high payoff application of the uncertainty-focused interpretation of the tradespace is in understanding high leverage points for technology insertion or investment. Technology development that results in more anticorrelated design options has the potential of reducing uncertainty and helping the system developer hedge against the risk of program failure due to correlated technology development. Of course, this hypothesis provides the basis for a whole new area of research.

10.7.3 Tradespace Exploration with Spiral Development

With spiral development, successive generations of system architecture and design result from each iteration of the spiral. Each iteration produces a learning opportunity that results in the refinement of the system architecture and design. A possible way to understand designs resulting from each spiral is through understanding the tradespace associated with each spiral (Derleth, 2003). Each spiral pass has a Pareto frontier, with some changes occurring in the Pareto Front with each spiral as new needs impact expectations around new capabilities. Thus, alternatives that were on the Pareto Front of the first spiral may not all be in the Pareto Front of the second spiral. In other words, some designs continue to offer the highest utility for a given cost, whereas others move to lower utility. This finding implies that designs are best chosen within the dynamic context of spirals. Sensitivities to hypothesized learning from one spiral to the next can be investigated in the tradespace context, resulting in superior architecture choices, especially where prior spiral decisions can limit future achievable capabilities.

10.7.4 Tradespace Exploration in Relation to Optimization and Decision Theory

For complex systems, value-driven tradespace exploration paradigm recognizes that concept design is a dynamic and complex process, and exploration is key to developing solutions that offer greater value to stakeholders. There are several reasons why system tradespace exploration is a preferred approach to optimization methods and decision-theory-only-based approaches such as utility functions (Table 10.3). While tradespace exploration can leverage both optimization and decision theory, the exploration paradigm encourages investigation of changing problem formulations and potential solutions and engagement with soft factors in order to iterate toward deeper understanding around the potential value of different alternatives.

	Tradespace Exploration	Optimization	Decision Theory
Objective function	Can handle changing, poorly defined objectives; can leverage decision theory and optimization	Require well-defined objectives	Employ utility functions to account for uncertainty in outcomes
Decision-maker preferences	Can handle changing preferences	Reflected in well-defined objective function	Can handle decision-maker preferences; changing preferences require revisiting decisions
Handling of “Soft” issues	Policy issues; system quality attributes such as flexibility and resilience	Not equipped to handle soft issues	Limited handling of soft issues
Uncertainty management and mitigation	Uncertainty–cost tradespace; uncertainty, mitigation through real options and decision rules (engineering view)	Limited uncertainty handling (stochastic optimization)	Uncertainty handling through real options (normative view) and expected value of perfect and imperfect information; use of expected utility; Bayesian networks
Visualization	Risk presentations; Kiviat charts; utility–cost functions; uncertainty cost functions; Pareto Front representation and change	Traditional visualization formats used by optimization methods	Utility curves; risk–benefit curves; sensitivity curves

Source: Adapted from Madni 2015c,d.

Trade-offs between design choices, perceived utility, and costs of alternatives are conveniently performed today using tradespace exploration. Tradespace exploration and visualization can support a variety of analysis and can be a source of much needed insights during the conceptual stage of system design. Tradespace analysis is complicated by changing needs and contexts over time. Today, tradespace exploration is being exploited: to identify promising designs; understand strengths and weaknesses of each design alternative across the tradespace; identify requirements and constraints that limit the ability to uncover less expensive solution; understand design sensitivities to changes in needs and context; and assess differential impacts of uncertainty on different alternative designs with opportunities for risk mitigation.

System architects and designers have the daunting challenge of evaluating multiple candidate design options, in terms of satisfying the needs of multiple stakeholders with potentially conflicting objectives while also contending with changes in stakeholders needs, emergence of new, relevant promising technologies, and uncertain geopolitical futures (Madni, 2015a). Fortunately, increases in computing power and decreasing cost of computing resources and wide availability of flexible tools have made it possible to more extensively explore the tradespace, generate more design options and CONOPS, and assess their feasibility.

In light of the forgoing, the first step is generating a tradespace for exploration. Figure 10.11 provides the overall concept for tradespace generation and exploration.

Figure 10.11 presents the tradespace generation and exploration system concept and functional architecture of a future system. As this figure shows, in the database generation phase, tradespace analysis is used to generate a large, representative number of candidates designs, concepts of operations, and their preanalyzed performance. The design and CONOPS are parameterized and expressed as parametric curves. System performance is expressed in terms of attributes and metrics of interest to the stakeholders. Computational and graphical tools allow the system engineers to query the database. Tradespace analysis employs utility–cost curves to concisely portray cost–benefit for each candidate design and CONOPS. Some of the key benefits of using this approach are as follows:

• Visualizing the immediate effects of changing needs
• Comparing candidate design alternatives (point designs) in a tradespace under a variety of contexts and conditions
• Readily identifying both functional and performance constraints on feasible solution set
• Visualizing different impacts of uncertainty across alternatives from tradespace
• Discerning alternatives in terms of their ability to change state (flexibility)
• Informing spiral acquisition
• Assessing policy robustness of alternative solutions

Tradespace exploration, a concept borrowed from decision analysis, is domain-independent. It explicitly links value propositions to alternatives, which may be obscured by politics, time, and misrepresentation. Tradespaces are constructed using concept-independent criteria (i.e., perceived benefits and costs) that allow comparison of distinct concepts based on the same tradespace.

Tradespaces can change over time. Therefore, tradespace exploration approaches need to be able to handle a dynamic tradespace. To this end, methods are needed that relax assumptions about static performance and constraints while adding the ability to take into account changing contexts and needs (epochs) over the short and long run (eras) using, for example, Epoch-Era Analysis (Ross, 2006; Ross & Rhodes, 2008; Roberts et al., 2009). In addition, there is a need to quantify the ability of the system to change (“changeability”) in the tradespace (Ross et al., 2008). This capability allows for the evaluation of a system's ability to reconfigure or be altered to respond to changing definitions of utility and cost over time. Several metrics have been defined to identify “good” designs in a time-varying context. These metrics fall into specific categories to achieve system value robustness, that is, maintain system value despite changing contexts and needs. These include highly changeable designs (filtered outdegree) and highly versatile or passively value robust designs (Pareto Trace, normalized Pareto trace, fuzzy normalized Pareto trace) (Ross et al., 2009a, 2009b). Temporal system properties that can be considered in tradespace exploration include system vulnerability and resilience (Ross et al., 2014).

10.8.1 Step 1: Determine Key Decision-Makers

The key decision-maker for this example was a representative of the government agency requesting the study. This representative provided the high-level operational needs statement for the system:

Provide maritime security for a particular littoral Area of Interest (AOI).

Additionally, the decision-maker expressed that stakeholders want a system that detects, identifies, and boards suspicious boats and that is capable of carrying out search and rescue missions.

10.8.2 Step 2: Scope and Bound the Mission

The goal of the MATE study is to determine what will be the “best” system when considering performance, ilities, and cost in this context. The decision-maker provided bounding constraints that said that the system must leverage inherited ground radar towers and existing manned patrol boats. It was suggested that the study consider UAVs as well. Additionally, the study was to consider the impact of loss of assets during the mission and means for mitigating the impact of such a loss.

10.8.3 Step 3: Elicit Attributes and Utilities (Preference Capture)

Discussions with the decision-maker elicited the function–objective–attribute decomposition shown in Figure 10.12.

For this study, the government agency acted as the actual decision-maker, desiring that both the security force and coast guard stakeholders be treated as separate “missions” to be satisfied. The goal for the government agency was to understand whether and what potential systems could satisfy each mission, as well as the overall cost objectives. Table 10.4 has an example list of attributes considered.

Through (lottery-based) interviews with the stakeholders, the following preferences were elicited:

1. Security Force

   Attribute Name	Weight	Min Acceptable (%)	Max Desired (%)
   Probability of detection	0.2	90	100
   Probability of identification	0.2	25	100
   Probability of intercept	0.6	10	100

   Figure 10.13 illustrates the single-attribute utility (SAU) curves for the Security Force mission showing increasing utility with increases in attribute. Of note in these curves is the “N=” at the top, which shows how constraining that SAU curve is on the tradespace, effectively showing the resulting fraction of the tradespace still feasible after application of that SAU curve. The asymmetry of the SAU curve is that designs are unacceptable (i.e., infeasible) if they score below U = 0, while they are not any more valuable, but are still feasible when scoring at or above U = 1. For this case, it is not possible to score higher than 100% probability, but in general, a design could conceivably score higher than the maximally desirable level of a particular attribute.

   

2. Coast Guard

   Attribute Name	Weight	Min Acceptable (%)	Max Desired (%)
   Probability of rescue	0.4	20	100
   Probability of detection	0.3	90	100
   Probability of identification	0.2	25	100
   Probability of intercept	0.1	10	100

   This example will focus on the Security Force mission stakeholder, so Coast Guard SAU curves are not included for brevity.

   The weights on the SAUs were elicited and determined to add to one, which coincides with a linear weighted sum multiattribute utility aggregation function:

   10.1

   where X_i is the value for attribute i, SAU_i is the single-attribute utility curve for attribute i, and k_i is the swing weight for attribute i.

   Note: the cost attributes were not rolled up into a utility function in order to help the decision-maker make explicit trades on cost versus benefit. This paradigm enables the decision-maker to explore and update his value model based on discoveries of what is possible and affordable.

10.8.4 Step 4: Define Design Vector Elements (Concept Generation)

Various brainstorming techniques were used to generate seven concepts for the system. These are summarized in Table 10.5.

These concepts were then deconstructed into a number of design variables that parameterize aspects of the system's form and CONOPs, described in Table 10.6.

The expectation is that not only does the form of the system matter (i.e., the number of each type of equipment), but also how it is operated (i.e., the CONOPs, including how tasks are assigned to vehicles).

10.8.5 Step 5: Develop Model(s) (Evaluation)

Once the attributes (outputs) and design variables (inputs) have been defined, the next step is to develop the evaluation model that will determine the attributes from each uniquely defined input concept design specified by enumerated design variable levels. For this study, the performance model was an agent-based discrete event simulator (Figure 10.14), which was developed to allow for analysis of emergent behaviors arising from the interaction of constituent systems and operational choices. Additionally, cost models and utility function evaluation models were used. The models were validated using input–output correlations as well as subject matter experts.

10.8.6 Step 6: Generate the Tradespace (Computation)

In this step, the design space is sampled, with continuous design variables sampled at discrete levels. This resulted in 10,368 unique design concepts to be evaluated in the model. The model then evaluates each point across the mission timeline. The particular mission being evaluated had a number of uncertain elements, including the exact timing and location of boats entering and leaving the AOI. These uncertain elements were treated as stochastic, with values for particular mission timelines determined via sampling of probability distributions. The repeated sampling of the stochastic variables resulted in a large number of different potential mission starting conditions. All design alternatives were exposed to the same set of potential missions. To keep this illustration simple, the results of the simulation are presented as the means of the output scores. This is a simplification for brevity and clarity. In practice, one could consider the full distributions and apply the uncertainty management techniques described earlier in the chapter (e.g., portfolio optimization or real options).

The reason for generating tradespace data rather than just a few point designs is because rich data sets can be explored to reveal complex relationships between design space and value space for generating intuition into a problem. This is especially important for systems that display emergence, have complex/conflicting stakeholder expectations, and are subject to changing contexts where previous experience may bias predictions.

10.8.7 Step 7: Explore the Tradespace (Analysis and Synthesis)

Exploration of the tradespace can take many forms, including using predefined workflows, such as question-driven exploration, or can be more freeform using various interactive visualization tools. For now we will investigate the attractive designs for the Security Force stakeholder, first starting with a point design, moving on to a set of points, and then on to full tradespaces.

As mentioned earlier, when interpreting these figures, the “N=” at the top indicates how many designs are considered currently feasible, based on constraints, including preferences (i.e., if a design point does not meet minimum acceptable utility, then it is rejected from the tradespace). The yield, which is the fraction of the originally evaluated tradespace that remains after constraints, is indicated as 82.81% of the original 10,368 evaluated alternatives.

Figure 10.15 illustrates a point design for the maritime security system, with indicated design variable values. In the absence of a tradespace, this design point appears reasonable, with an acquisition cost around $40M and delivering a utility of 0.56. Further investigation reveals that this design has a probability of detection of 100%, probability of identification of 89.5%, and a probability of intercept of 59.5%. It is unclear from looking at the point alone what the trade-offs might be in terms of performance versus cost, for example.

Moving on to identifying an efficient set of points, Figure 10.16 illustrates a set of Pareto frontier points (efficient in terms of Security MAU and Acquisition Cost). All of these designs have in common a central command authority and two operators per UAV. Table 10.7 illustrates the attribute scores, acquisition cost, and MAU for these points. The general trend is that the increased cost moving from A to F gets you more performance in terms of probability of intercept. This coincides with having additional Hermes UAVs, as well as manned patrol boats.

Taking the exploration further results in the tradespace shown in Figure 10.17, with all 10,384 evaluated points considered, of which 82.81% are feasible. Highlighted is the full evaluated Pareto frontier set, expanding the alternatives for consideration in terms of most efficient MAU for Acquisition Cost. Using color-by-design variable, we can investigate design choice impacts on MAU and cost, expanding the scope of insight beyond the handful of Pareto points considered previously.

Figure 10.18 shows the tradespace colored by the number of Hermes and number of Shadows, respectively. The pattern of increases in the number of Hermes in the Pareto points is here again repeated, with three apparent groupings of increases in cost and utility for increases in Hermes. The utility increase from the increase in the number of Shadows is not as clear.

Figure 10.19 shows the tradespace colored by the number of manned patrol boats and type of command authority, respectively. The three groupings in the tradespace are clearly driven by the number of manned patrol boats. This corresponds to their role in boarding suspicious boats, strongly driving MAU (and cost) with increases in number. Central command authority mostly dominates distributed authority in the tradespace.

Figure 10.20 shows the tradespace colored by number of operators per UAV and the number of geographic zones. The apparent general increase in MAU with the two operators over the 0.25 operators per UAV appears counterintuitive (i.e., having each operator control more than one UAV seems as a force multiplier and therefore increases the performance of the system). Upon deeper inspection, we found that cognitive overloading leads to decreased operator performance when each operator must control multiple UAVs. This emerged in the performance model as each operator manages his own queue of tasking during the mission. Task backlogs accumulate as the operators switch between UAVs, operating at a constrained pace. As for number of geographic zones, this reflects the CONOPs trade of whether to allow the vehicles to travel the entire AOI or to constrain them to a fixed zone. In the lower cost region of the tradespace (where designs have fewer vehicles), the highest utility points tend to have one zone, whereas the higher cost region (and larger number of vehicles) the highest utility points tend to have two zones. These reflect the general need to have vehicles evenly distributed to increase the chances that they can respond quickly to a target of interest before it moves away.

Investigating a larger tradespace both confirms and expands upon the insights generated through investigation of a handful of design points. At some point, however, additional insights gained can be expected to diminish with increases in the size of the tradespace. The most appropriate size for a particular study should be determined on a case-by-case basis, with the view that tradespaces should scale up, rather than down. As noted previously in this book, the insights gained are only as good as the model developed and the quality of inputs. Through iterative exploration of the tradespace, model goodness can be determined. Tradespace exploration encourages broad sensitivity analysis of both models and preference inputs to better generate insights into trend relationships between what is desired (value proposition) and what is possible (evaluated design alternatives).