5.3.1 Frame Decision

The first step of the decision management process is to frame the decision and to tailor the decision process. To help ensure that the decision-makers and stakeholders fully understand the decision context and to enhance the overall traceability of the decision, the systems engineer should capture a description of the system baseline as well as a notion for how the envisioned system will be used (concept of operations) along with system boundaries and anticipated interfaces. Decision context includes such details as the timeframe allotted for the decisions, an explicit list of decision-makers and stakeholders, available resources, and expectations regarding the type of action to be taken as a result of the decision at hand as well as decisions anticipated in the future (Edwards et al. 2007). The best practice is to identify a decision problem statement that defines the decision in terms of the system life cycle. Next, three categories of decisions should be listed: decisions that have been made, decisions to be made now, and subsequent decisions that can be made later in the life cycle. Effort is then focused on the decisions to be made now.

Once the decision at hand is sufficiently framed, systems engineers must select the analytical approach that best fits the frame and structure of the decision problem at hand. For deterministic problems, optimization models can explore the decision space. However, when there are “… clear, important, and discrete events that stand between the implementation of the alternatives and the eventual consequences…” (Edwards et al., 2007), a decision tree is a well-suited analytical approach, especially when the decision structure has only a few decision nodes and chance nodes. As the number of decision nodes and chance nodes grow, the decision tree quickly becomes unwieldy and loses some of its communicative power. However, decision trees and many optimization models require consequences to be expressed in terms of a single number. This is commonly accomplished for decision situations where the potential consequences of alternatives can be readily monetized and end state consequences can be expressed in dollars, euros, yen, and so on. When the potential consequences of alternatives within a decision problem cannot be easily monetized, an objective function can often be formulated to synthesize an alternative's response across multiple, often competing, objectives. A best practice for this type of problem is the multiple objective decision analysis (MODA) approach (Chapter 2).

The decision management method most commonly employed by systems engineers is the trade study and more often than not employs some form of MODA approach. The aim is to define, measure, and assess shareholder and stakeholder values and then synthesize this information to facilitate the decision-maker's search for an alternative that represents the optimally balanced response to often competing objectives. Major system projects often generate large amounts of data from many separate analyses performed at the system, subsystem, component, or technology level by different organizations. Each analysis, however, only delivers one dimension of the decision at hand, one piece of the puzzle that the decision-makers are trying to assemble. These analyses may have varying assumptions and may be reported as standalone documents, from which decision-makers must somehow aggregate system-level data for all alternatives across all dimensions of the tradespace in his or her head. This would prove to be an ill-fated task as all decision-makers and stakeholders have cognitive limits that preclude them from successfully processing this amount of information in their short-term memory (Miller 1956). When faced with a deluge of information that exceeds human cognitive limits, decision-makers may be tempted to oversimplify the tradespace by drastically truncating objectives and/or reducing the set of alternatives under consideration, but such oversimplification runs a high risk of generating decisions that lead to poor outcomes.

By providing techniques to decompose a trade-off decision into logical segments and then synthesize the parts into a coherent whole, a formal decision management process offers an approach that allows the decision-makers to work within human cognitive limits without oversimplifying the problem. In addition, by decomposing the overall decision problem into smaller elements, experts can provide assessments of alternatives as they perform within the objective associated with their area of expertise. Buede and Choisser put it this way,

These component parts can be subdivided as finely as needed so that the total expertise of the system design team can be focused, in turn, on specific, well-defined issues. The analyses on the component parts can then be combined appropriately to achieve overall results that the decision makers can use confidently. The benefits to the decision maker of using this approach include increased objectivity, less risk of overlooking significant factors and, perhaps most importantly, the ability to reconstruct the selection process in explaining the system recommendation to others. Intuition is not easily reproducible.

(Buede & Choisser 1992)

MODA approaches generally differ in the techniques used to elicit values from stakeholders, the use of screening techniques, the degree to which an alternative's responses to objectives (and subobjectives) are aggregated, the mathematics used to aggregate such responses, the treatment of uncertainty, the robustness of sensitivity analyses, the search for improved alternatives, and the versatility and quality of tradespace visualization outputs. If time and funding allow, systems engineers may want to conduct trade-off studies using several techniques, compare and contrast results, and reconcile any differences to ensure that the findings are robust. Although there are many possible ways to specifically implement MODA, the discussion contained in the rest of this chapter represents a short summary of best practices.

5.3.2 Develop Objectives and Measures

Defining how a decision will be made may seem straightforward, but often becomes an arduous task of seeking clarity amidst a large number of ambiguous stakeholder need statements. The first step is to use the information obtained from the Stakeholder Requirements Definition Process, Requirements Analysis Process, and Requirements Management Processes to develop objectives and measures. If these processes have not been started, then stakeholder analysis is required. Often, this begins with reading documentation on the decision topic followed by a visit to as many decision-makers and stakeholders as reasonable and facilitating discussion about the decision problem. This is best done with interviews and focus groups with subject matter experts and stakeholders.

For systems engineering trade-off analyses, top-level stakeholder value often includes competing objectives of performance, development schedule, life cycle costs, and long-term viability. For corporate decisions, shareholder value would be added to this list. With the top-level objectives set, lower levels of objective hierarchy should be discovered. For performance-related objectives, it is often helpful to work through a functional decomposition (usually done as part of the requirements and architectural design processes) of the system of interest to generate a thorough set of potential objectives. Start by identifying inputs and outputs of the system of interest and craft a succinct top-level functional statement about what the system of interest does, identifying the action performed by the system of interest to transform the inputs into outputs. Test this initial list of fundamental objectives for key properties by checking that each fundamental objective is essential and controllable and that the set of fundamental objectives is complete, nonredundant, concise, specific, and understandable (Edwards et al. 2007). See Figure 5.4 for a list of the key properties for a set of fundamental objectives.

Beyond these best practices, the creation of fundamental objectives is as much an art as it is a science. This part of the decision process clearly involves subjectivity. It is important to note, however, that a subjective process is not synonymous with an arbitrary or a capricious process. As Keeney points out,

Subjective aspects are a critical part of decisions. Defining what the decision is and coming up with a list of objectives, based on one's values, and a set of alternatives are by nature subjective processes. You cannot think about a decision, let alone analyze one, without addressing these elements. Hence, one cannot even think about a decision without incorporating subjective aspects

(Keeney 2004)

The output of this process step takes on the form of a fundamental objectives hierarchy as illustrated in Figure 5.5.

For completeness, it is often helpful to build a crosswalk between the objectives hierarchy and any stakeholder need statements or capability gap lists as illustrated in Table 5.2. This activity helps ensure that all stakeholder need statements or capability gaps have been covered by at least one objective. It also aids in identifying objectives that do not directly trace to an expressed need. Such objectives will need additional explanation to justify their inclusion in the hierarchy. It should be noted, however, that it is common to have such objectives included in a hierarchy because stakeholders are often silent about needs that are currently satisfied by the incumbent system but would not be happy if, in an effort to fill a perceived need, the new system created a new gap: for example, up-armored vehicles that become unreliable or insufficiently mobile.

For each fundamental objective, a measure (also known as attribute, criterion, and metric) must be established so that alternatives that more fully satisfy the objective receive a better score on the measure than those alternatives that satisfy the objective to a lesser degree. Table 5.3 illustrates this one-to-one mapping of objective and measure.

A measure should be unambiguous, comprehensive, direct, operational, and understandable (Keeney & Gregory 2005). Table 5.4 defines these properties of a high-quality measure.

Source: Data from Keeney & Gregory 2005.

Keeney has identified three types of measures – natural measure (kilometers, degrees, probability, seconds, etc.), constructed measure (Dow Jones Industrial Average, Heat Index, Consumer Price Index, etc.), and a proxy measure (usually a natural measure of a consequence that is thought to be correlated with the consequence of interest). Keeney recommends natural measures whenever possible since they tend to be commonly used and very understandable. When a natural measure is not available to describe a consequence, he recommends a constructed measure that directly describes the consequence of interest. If neither a natural measure nor a constructed measure is practical, then a proxy measure using a natural scale is often workable although by definition, it is an indirect measure as it does not directly describe the consequence of interest.

A defining feature of Multiobjective Decision Analysis (also called multiattribute value theory) is the transformation from measure space to value space that enables mathematical representation of a composite value score across multiple measures. This transformation is performed through the use of a value function. Value functions describe returns to scale on the measure. In other words, value functions describe the degree of satisfaction stakeholders perceive at each point along the measure scale.

There are several techniques available to elicit value functions and priority weightings from stakeholders. One of the more popular techniques used in marketing circles is Conjoint Analysis (Green et al., 2001) where stakeholders are asked to make a series of pairwise comparisons between hypothetical products. For decisions that involve fewer than eight competing objectives, Conjoint Analysis is a compelling technique for generating representative value schemes. As an objectives hierarchy grows beyond eight objectives, the number of pairwise comparisons required to formulate the representative value schemes balloons to an unreasonable number. Since many complex systems engineering decision tasks tend to involve the balancing of 25 to 35 objectives, the value scheme elicitation approach described in this chapter is a direct interview value function formulation technique coupled with a swing weight matrix methodology for determining priority weightings.

When creating a value function, one ascertains whether stakeholders believe there is a walk-away point on the objective measure scale (x-axis) and map it to 0 value on the value scale (y-axis). A walk-away point is defined as the measure score where regardless of how well an alternative performs in other measures, the decision maker will walk away from the alternative. Working with the stakeholder, find the measure score beyond which an alternative provides no additional value, label it “meaningful limit” (also called ideal), and map it to 100 (1 and 10 are also common scales) on the value scale (y-axis). If the returns to scale are linear, connect the walk-away value point to the meaningful value point with a straight line. If there is reason to believe stakeholder value behaves with nonlinear returns to scale, pick appropriate inflection points and draw the curve. The rationale for the shape of the value functions should be documented for traceability and defensibility (Parnell et al., 2011). Figure 5.6 provides examples of some common value function shapes.

Practice suggests that eliciting two end points and three inflection points provides informative value functions without overtaxing the systems engineering or stakeholder.

• Walk Away: Stakeholder will dismiss an alternative if it fails to meet at least this level regardless of how it performs on other value measures (1 point for meeting, 0 points if missed).
• Marginally Acceptable: Stakeholder begins to become interested, and beyond this point the perceived value increases rapidly (10 points).
• Target: Desired level (50 points).
• Stretch Goal: Improving beyond this point is considered gold plating, so there is very little available value between this point and meaningful limit (90 points).
• Meaningful Limit: Theoretical limit or known practical limit beyond which would be considered unrealistic (100 points).

In an effort to capture the voice of the customer, systems engineers will often ask a stakeholder focus group to prioritize their requirements. As Keeney puts it,

Most important decisions involve multiple objectives, and usually with multiple-objective decisions, you can't have it all. You will have to accept less achievement in terms of some objectives in order to achieve more on other objectives. But how much less would you accept to achieve how much more?

(Keeney 2002)

The mathematics of Multiobjective Decision Analysis (MODA) requires that the weights depend on importance of the preferentially independent measure and the range of the measure (walk away to stretch goal or ideal). A useful tool for determining weightings is the swing weight matrix. (Parnell et al., 2011) For each measure, consider its importance by determining if the measure corresponds to a defining capability, a critical capability, or an enabling capability and also consider the variation measure range by considering the gap between the current capability and the desired capability and put the name of the measure in the appropriate cell of the matrix. Swing weights are then assigned to each measure according to the required relationship rules described in Figure 5.7. Swing weights are then converted to measure weights by normalizing such that the set sums to 1. For the purposes of swing weight matrix use, consider a defining capability to be one that directly traces to a verb/noun pair identified at the top-level (level 0) functional definition of the system of interest – the reason why the system exists. Consider enabling capabilities to trace to functions that are clearly not the reason why the system exists but allow the core functions to be executed more fully. Let critical capabilities be those that are more than enabling but not quite defining.

All decisions involve elements of subjectivity, the distinctive feature of formal decision management process is that these subjective elements are rigorously documented so that the consequences can be identified and assessed. Toward this end, it is considered good practice to document the measured, the value function, and the priority weighting along with associated rationale for each fundamental objective.

5.3.2.1 Example of Developing Objectives and Measures

Returning to the sUAV case study example, we find that after some quality time with many of the stakeholders and hours digging through white papers, memos, and presentations relevant to the problem statement, the lead systems engineer spearheaded a requirements analysis effort that included a functional decomposition. This exercise helped the systems engineering trade-off study team understand and be able to articulate what the system of interest is expected to “do,” which in turn enabled them to construct the functional performance objectives shown in the objectives hierarchy shown in Figure 5.8. Note that stakeholder value is measured in terms of not only functional performance but also life cycle costs and development schedule. (Although not addressed here due to space considerations, the notion of long-term viability fits well within an objectives hierarchy such as this).

With the objectives hierarchy in hand, the study team identified measures for each of the functional performance objectives as shown in Table 5.5.

1.1 Be Soldier Transportable	1.1.1 Avoid Impeding Soldier Endurance	Measure: % decrease in sustainable march speed
	1.1.2 Avoid Impeding Soldier Sprint	Measure: % increase in soldier sprint time
	1.1.3 Avoid Impeding Soldier Jump	Measure: % degredation in soldier jump height
1.2 Maneuver to and Dwell at Area of Interest	1.2.1 Reach Areas of Interest Quickly	Measure: Max flight speed (km/hour)
	1.2.2 Reach Distant Areas of Interest	Measure: Maximum operational range (km)
	1.2.3 Dwell @ Area of Interest for Extended Periods	Measure: Operational Endurance (hours)
1.3 Collect ISR Info	1.3.1 Be Responsive to a Variety of ISR Data Requests	Measure: ISR Data Request Responsiveness Index
	1.3.2 Collect High-Quality Imagery During Daytime	Measure: TTP rating per NV-IPM @ 3000m full light
	1.3.3. Collect High-Quality Imagery at Night	Measure: TTP rating per NV-IPM @ 3000m low light
	1.3.4 Collect High-Quality Imagery in Obscured Env.	Measure: TTP rating per NV-IPM @ 3000m w/ smoke
1.4 Securely Exchange Info w/ Command Station	1.4.1 Exchange Info Across Terrains & Geometries	Measure: BLOS comms capable (yes/no)
	1.4.2 Send Large Volumes of Data Quickly & Reliably	Measure: High data rate payload comm link? (Y/N)
	1.4.3 Avoid Spoofing, Jamming, Intercept	Measure: Digital C2 link? (Y/N) Digital Payload Com link? (Y/N)
1.5 Be Recoverable & Tamper Resistant	1.5.1 Enable High Probability of Recovery	Measure: Subjective assessment of landing scheme
	1.7.2 Render System Useless Upon Enemy Capture	Measure: Command self destruct feature?

Not shown in Table 5.5 are the measures for the Life Cycle Cost objective and the Development Schedule objective. These two measures are discussed here. The life cycle cost measure for this hypothetical case study is the sum of the rough order of magnitude estimates for development costs, procurement costs, training costs, maintenance costs, and wartime costs. Schedule duration for this exercise is measured as the number of years that the development effort requires estimated at the 80% confidence level after considering the uncertainty associated with the duration estimates for each configuration item to mature from its current state to form, fit, and function tested across temperatures plus the estimate for time required for system-level integration and test.

With a good understanding of how each objective is to be measured, the lead systems engineer worked to understand the degree of satisfaction that each stakeholder perceives at each point along a particular measure scale and then expressed these relationships as a set of value functions. To accomplish this, the lead systems engineer of the sUAV effort worked with a small group of stakeholders to document a walk-away point, a marginally acceptable point, a target point, a stretch goal, and a meaningful limit. The lead systems engineer repeated this process for several stakeholder groups in order to capture any differences among value schemes. The lead systems engineer maintained a record of each set of value functions so that he may use them as part of the sensitivity analysis later in the process. Table 5.6 describes the value functions associated with the 17 measures of this hypothetical sUAV case study as constructed by one of the stakeholder groups. Notice that life cycle cost measures and schedule duration measures are not included in Table 5.6 due to space considerations. Figure 5.9 provides a graphical view of 3 of the 17 value functions.

Name	Value	Functional Performance
		Be Transported			Fly			Collect				Communicate			End
		Avoid Impeding Soldier Endurance	Avoid Impeding Soldier Sprint	Avoid Impeding Soldier Jump	Reach Area of Interest (10 km) Quickly	Reach Distant Areas of Interest	Dwell at Area of Interest	Be Responsive to a Variety of ISR Data Requests	Collect High Quality Imagery During Day	Collect High Quality Imagery During Night	Collect High Quality Imagery In Obscured Environments	Exchange Info Across Various Terrains & Geometries	Send ISR Imagery Quickly and Reliably	Avoid Spoofing, Jamming, or Communicate Intercept	Enable High Probability of Recovery	Render System Useless Upon Enemy Capture
		%	%	%	min	km	hrs	VI	Pd	Pd	Pd	L/B	sec	A/D	%	y/n
Walk-away	1	10	15	20	15	10	1	1	0.1	0.1	0.1	−	90	−	90	−
Marginally Acceptable	10	8	12	16	12	15	2	2	0.2	0.2	0.2	L	30	A	92	N
Target	50	5	7.5	10	8	30	4	5	0.7	0.7	0.7	−	10	−	95	−
Stretch goal	90	2	3	4	4	45	8	8	0.9	0.9	0.9	B	5	D	98	Y
Meaningful limit	100	0	0	0	1	50	16	10	1.0	1.0	1.0	−	1	−	100	−

To complete his understanding of the stakeholder value, the lead systems engineer set out to identify the objectives of which the stakeholders were willing to accept marginal returns in order to achieve high returns on others. By working through a swing weight matrix with each stakeholder group, the lead systems engineer identified weightings for each measure. The normalized weights for one of the stakeholder groups are depicted in Figure 5.10. Weights developed by other stakeholder groups were also documented and maintained for use in the sensitivity analysis of later steps.

With the value schemes of stakeholders having been captured, the lead systems engineer knows how goodness will be measured for each alternative considered within the systems engineering trade-off analysis. The lead systems engineer can now turn his attention to generating sUAV system alternatives. The next section of this chapter describes best practices for generating creative alternatives and is followed by the continuation of the sUAV case study.

5.3.3 Generate Creative Alternatives

For many trade studies, the alternatives will be systems composed of many interrelated subsystems. It is important to establish a meaningful product structure for the system of interest and to apply this product structure consistently throughout the decision process effort in order to aid effectiveness and efficiency of communications about alternatives. The product structure should be a useful decomposition of the physical elements of the system of interest.

Each alternative is composed of specific design choices for each generic product structure element. The ability to quickly communicate the differentiating design features of given alternatives is a core element of the decision-making exercise. Tables 5.8–5.10 provide a template for succinct yet complete system-level alternative descriptions. These subsystem design choices have system-level consequences across the objectives hierarchy. Every subsystem design choice will impact system-level cost, system-level development schedule, and system-level performance. It is important to emphasize that these design choices are not fundamental objectives, they are means objectives important only to the degree that they assist in achieving fundamental objectives. It may be useful to think of design choices as the levers used by the system architect to steer the system design toward a solution that best satisfies the all elements of stakeholder value – the full fundamental objectives hierarchy. These levers are very important and care should be given in this step of the process to clearly and completely identify specific design choices for each generic product structure element for every alternative being considered. Incomplete or ambiguous alternative descriptions can lead to incorrect or inconsistent alternative assessments in the process described later. The ability to quickly and accurately communicate the differentiating design features of given alternatives is a core element of the decision-making exercise.

System characteristics are interesting consequences of subsystem design choices measured at the system level but are not themselves fundamental objectives. For example, the weight of a system alternative is classified as a system characteristic in that weight is neither a design decision nor a fundamental objective. Weight is not a design decision but rather a consequence of all the subsystem design choices made throughout the product structure. Weight is not a fundamental objective because it is not inherently good or bad although it does factor into many fundamental objectives measures. Documenting characteristics is considered a best practice for two reasons;

1. if senior stakeholders are known to ask about certain aspects of various alternatives, the lead systems engineer should be prepared to provide an immediate answer, and
2. some system characteristics are part of so many objective measures that it is useful to report them at the same level as system design choices for the sake of clarity and explanatory power.

5.3.3.1 Example of Generating Creative Alternatives

Returning to our sUAV example, the lead systems engineer has identified the following four top-level elements of the sUAV physical architecture: the Air Vehicle, the ISR Collecting Payload, the Communication Links, and the Ground Elements. The sUAV Physical Architecture Description in Table 5.7 decomposes the four top level physical elements into generic subelements and also provides a list of specific design choices available for each generic subelement of the physical architecture.

Air Vehicle
Propulsion System	Energy Source	Prop Size & Location	Wing Span	Wing Config.	Fin Config.	Actuators	Airframe Material	Autopilot	Launch	Land
Electric 300W	Li-Ion Battery	18” Rear	4 ft	Conv.	Twin Boom Conv.	Electro-magnetic	Graphite Epoxy	Preprogram, Auto	Hand	Skid and Belly
Electric 600W	Li-S Battery	22” Rear	5 ft	Canard	Inverted V	Hydraulic	Aramid Epoxy	Semiauto	Tensioned Line	Net
Piston Engine 2.5HP	Fuel Cell	26” Rear	6 ft	Tandem Wing	V Tail	MEMS	Boron Epoxy	Remotely Piloted	Gun Launch	Parachute
Piston Engine 4.0HP	Solar	18” Front	7 ft	Three Surface	H Tail		Fiberglass Epoxy			Deep Stall
	JP-8 Fuel	22” Front	8 ft		Cruciform
		26” Front	9 ft
Fixed	None	None	Small Fixed Antenna transmits analog data direct to GCS (VHF or UHF)	Small Fixed Antenna transmits analog data direct to GCS (VHF or UHF)	Dipole	Ruggedized Laptop	Keyboard	Generator
Pan–tilt	4 Megapixel Daylight Camera	Cooled 320 × 240 MWIR	Small, Fixed, Nonpointing Antenna transmits digital data to LEO Satellite (L Band)	Small, Fixed, Nonpointing Antenna transmits digital data to LEO Satellite (L Band)	Parabolic Reflector	Wearable Computer	Joystick	Battery + Generator
Roll–tilt	8 Megapixel Daylight Camera	Cooled, 640 × 480 MWIR		Mech. Steerable parabolic dish transmits digital data to GEO Satellite (Ka or Ku Band)		Smartphone	Touchscreen	Battery + Backup Batteries
Pan–tilt–roll		Cooled 1280 × 720 MWIR & LWIR		Electronically Steered Phased Array Antenna transmits digital data to GEO Satellite (Ka or Ku Band)			Stylus
		Uncooled 1024 × 768 MWIR & LWIR

Using the Physical Architecture Description in Table 5.6, the lead systems engineer and his study team developed the 12 system-level sUAV concepts in Tables 5.8–5.10. The team used the table format for describing the alternatives to ensure that each system would be described completely, succinctly, and consistently to aid in the efficiency and effectiveness of communications throughout the trade study.

	1	2	3	4
	Buzzard I	Buzzard II	Cardinal I	Cardinal II
Subsystem/Component	Design Choice	Design Choice	Design Choice	Design Choice
Air vehicle
Propulsion System	Electric 300W	Electric 300W	Electric 300W	Electric 300W
Energy Source	Li-Ion Battery	Li-Ion Battery	Li-S Battery	Li-S Battery
Prop Size and Location	18” Rear	18” Rear	20” Rear	20” Rear
Wing Span	5'	5'	6'	6'
Wing Configuration	Canard	Canard	Conventional	Conventional
Fin Configuration	Inverted V	Inverted V	Twin Boom	Twin Boom
Actuators	Electromagnetic	Electromagnetic	Electromagnetic	Electromagnetic
Airframe Material	Graphite Epoxy	Graphite Epoxy	Graphite Epoxy	Graphite Epoxy
Autopilot	Semiauto	Semiauto	Remotely Piloted	Remotely Piloted
Launch Mechanism	Hand	Hand	Hand	Hand
Landing Mechanism	Belly	Belly	Belly	Belly
ISR Collecting Payload
Sensor Actuation	Fixed	Fixed	Fixed	Fixed
EO Imager	4 MP	4 MP	4 MP	4 MP
IR Imager	320 × 240 MWIR	640 × 480 MWIR	320 × 240 MWIR	640 × 480 MWIR
Communication Links
Command and Control Link	Fixed VHF	Fixed VHF	Fixed VHF	Fixed VHF
Payload Data Link	Fixed VHF	Fixed VHF	Fixed VHF	Fixed VHF
Ground Elements
Antenna	Dipole	Dipole	Dipole	Dipole
Computer	Laptop	Laptop	Smartphone	Smartphone
User Input Device	Keyboard	Keyboard	Joystick	Joystick
Power	Battery + Spare	Battery + Spare	Battery + Spare	Battery + Spare

	5	6	7	8
	Crow I	Crow II	Pigeon I	Pigeon II
Subsystem/Component	Design Choice	Design Choice	Design Choice	Design Choice
Air Vehicle
Propulsion System	Electric 600W	Electric 600W	Electric 600W	Electric 600W
Energy Source	Li-Ion Battery	Li-Ion Battery	Li-S Battery	Li-S Battery
Prop Size and Location	22” Rear	22” Rear	20” Rear	20” Rear
Wing Span	6'	6'	6'	6'
Wing Configuration	Tandem Wing	Tandem Wing	Conventional	Conventional
Fin Configuration	V Tail	V Tail	Twin Boom	Twin Boom
Actuators	MEMS	MEMS	Electromagnetic	Electromagnetic
Airframe Material	Graphite Epoxy	Graphite Epoxy	Graphite Epoxy	Graphite Epoxy
Autopilot	Semiauto	Semiauto	Remotely Piloted	Remotely Piloted
Launch Mechanism	Hand	Hand	Hand	Hand
Landing Mechanism	Belly	Belly	Belly	Belly
ISR Collecting Payload
Sensor Actuation	Pan–tilt	Pan–tilt	Pan–tilt	Pan–tilt
EO Imager	8 MP	8 MP	8 MP	8 MP
IR Imager	1280 × 720 MWIR & LWIR cooled	1280 × 720 MWIR & LWIR uncooled	1280 × 720 MWIR & LWIR cooled	1280 × 720 MWIR & LWIR uncooled
Communication Links
Command and Control Link	Fixed VHF	Fixed VHF	Fixed VHF	Fixed VHF
Payload Data Link	Fixed VHF	Fixed VHF	Phased Array Ka	Phased Array Ka
Ground Elements
Antenna	Dipole	Dipole	Dipole & Dish	Dipole & Dish
Computer	Laptop	Laptop	Laptop	Laptop
User Input Device	Keyboard	Keyboard	Joystick	Joystick
Power	Battery + Spare	Battery + Spare	Battery + Spare	Battery + Spare

	9	10	11	12
	Robin I	Robin II	Dove I	Dove II
Subsystem/Component	Design Choice	Design Choice	Design Choice	Design Choice
Air Vehicle
Propulsion System	Piston 2.5 HP	Piston 2.5 HP	Piston 4.0 HP	Piston 4.0 HP
Energy Source	JP-8	JP-8	JP-8	JP-8
Prop Size and Location	26” Front	26” Front	28” Front	28” Front
Wing Span	8'	8'	9'	9'
Wing Configuration	Conventional	Conventional	Conventional	Conventional
Fin Configuration	H Tail	H Tail	Cruciform	Cruciform
Actuators	Hydraulic	Hydraulic	Hydraulic	Hydraulic
Airframe Material	Fiberglass Epoxy	Fiberglass Epoxy	Fiberglass Epoxy	Fiberglass Epoxy
Autopilot	Remotely Piloted	Remotely Piloted	Remotely Piloted	Remotely Piloted
Launch Mechanism	Tensioned Line	Tensioned Line	Tensioned Line	Tensioned Line
Landing Mechanism	Net	Net	Net	Net
ISR Collecting Payload
Sensor Actuation	Pan–tilt	Pan–tilt	Pan–tilt	Pan–tilt
EO Imager	8 MP	8 MP	8 MP	8 MP
IR Imager	1280 × 720 MWIR & LWIR cooled	1280 × 720 MWIR & LWIR uncooled	1280 × 720 MWIR & LWIR cooled	1280 × 720 MWIR & LWIR uncooled
Communication Links
Command and Control Link	Fixed VHF	Fixed VHF	Fixed VHF	Fixed VHF
Payload Data Link	Elect. Steered Phased Array Ka	Elect. Steered Phased Array Ka	Mech. Steered Dish Ka	Mech. Steered Dish Ka
Ground Elements
Antenna	Dipole	Dipole	Dipole and Dish	Dipole and Dish
Computer	Laptop	Laptop	Laptop	Laptop
User Input Device	Keyboard	Keyboard	Joystick	Joystick
Power	Battery + Gen.	Battery + Gen.	Battery + Gen.	Battery + Gen.

The lead systems engineer knows that with the alternatives defined, he is ready to start collecting data regarding each system's response to the measures established earlier in the process. The next section of this chapter walks through some best practices with regard to assessing alternatives via deterministic analysis followed by the continuation of this sUAV example.

5.3.4 Assess Alternatives via Deterministic Analysis

With objectives and measures established and alternatives identified and defined, the decision team should engage subject matter experts, ideally equipped with operational data, test data, models, simulations and expert knowledge. Often a mapping between physical architecture elements to fundamental objectives may help identify the types of subject matter expertise needed to fully assess each alternative against a particular objective (Table 5.11 – - Physical Architecture to Fundamental Objective Mapping). These simple maps often grow to more complex flow diagrams showing the interrelationships between physical architecture choices, different levels of intermediate measures, and, finally, the fundamental objectives.

It may be helpful to expand these simple maps into more informative Assessment Flow Diagrams (AFDs) that trace the relationships between physical means, intermediate measures, and fundamental objectives. As an example of such a diagram, consider the sample provided in Figure 5.11. An Assessment Flow Diagram helps individual subject matter experts understand how their area of expertise fits into the larger assessment picture, from where inputs to feed their particular model will be coming and to where their outputs will be consumed. An AFD can be used by the lead systems engineer to organize, manage, and track assessment activities especially when used in conjunction with the consequence scorecard shown in Table 5.12 and Table 5.13.

In addition to the organization and communication benefits, an AFD seems to provide some psychological benefits to the subject matter experts (SMEs) conducting the assessments and to the stakeholders hoping to make use of the results. An AFD gives an SME confidence that their analysis will not be ignored and sends the message that their expertise is important and needed and their results will find their way to the decision table in proper context, as a piece of the whole assessed in terms meaningful to the stakeholder. Similarly, by showing the pedigree of the data feeding the decision support model, an AFD gives the stakeholders confidence that the trade-off analysis rests on a solid foundation and not a product of generalists sitting around the table voting.

The decision team can prepare for subject matter expert engagement by creating structured scoring sheets. Assessments of each concept against each criterion can be captured on separate structured scoring sheets for each alternative/measure combination. Each score sheet contains a summary description of the alternative under examination and a summary of the scoring criteria to which it is being measured. The structured scoring sheet should contain ample room for the evaluator to document the assessed score for the particular concept against the measure followed by clear discussion providing the rationale for the score, noting how design features of the concept under evaluation led to the score as described in the rating criteria. Whenever possible, references to operational data, test data, calculations, models, simulations, analogies, or experience that led to a particular score should be documented.

Creating separate structured scoring sheets for each alternative/measure combination may become somewhat cumbersome for large studies. In practice, a separate structured scoring sheet is often constructed for each measure only and alternatives are identified as separate rows within each measure sheet. Table 5.12 provides a sample format for such a scoring sheet. This approach has the added benefit of reinforcing the notion that each alternative is assessed using the same measure and reducing the risk of inconsistent assessments.

After all the structured scoring sheets have been completed for each alternative/measure combination, it is useful to summarize all the data in tabular form. Each column in such a table would represent a measure and each row would represent a particular alternative. Table 5.13 provides a sample structure identified here as a consequences scorecard.

5.3.4.1 Example of Assessing Alternatives via Deterministic Analysis

Continuing the sUAV case study example, the lead systems engineer recruited a team of subject matter experts to assess each alternative against a measure that aligns with their skill set. For instance, human factors experts assessed impact to soldier mobility and endurance, aerospace engineers assessed the measures pertaining to flight, electrical engineers specializing in sensors assessed the ISR data collection measures, communication engineers scored the information transit and receive measures, and mechanical engineers rated each alternative against the recover measures. Each subject matter expert was provided with a scoring sheet for recording their findings. The lead systems engineer took the findings from the scoring sheets and created the consequence scorecard shown in Table 5.14.

	Functional Performance
			Be Transported			Fly			Collect				Communicate			End
			Avoid Impeding Soldier Endurance	Avoid Impeding Soldier Sprint	Avoid Impeding Soldier Jump	Reach Area of Interest (10 km) Quickly	Reach Distant Areas of Interest	Dwell at Area of Interest	Be Responsive to a Variety of ISR Data Requests	Collect High Quality Imagery During Day	Collect High Quality Imagery During Night	Collect High Quality Imagery In Obscured Environments	Exchange Info Across Various Terrains & Geometries	Send ISR Imagery Quickly and Reliably	Avoid Spoofing, Jamming, or Communicate Intercept	Enable High Probability of Recovery	Render System Useless Upon Enemy Capture	Life Cycle Costs	Development Schedule Duration
ID	Name	Image	%	%	%	min	km	hrs	VI	Pd	Pd	Pd	LB	sec	AD	%	y/n	$B	yrs
1	Buzzard I		1	2	3	15	10	1	2	0.3	0.2	0.1	L	30	A	90	90	1	2
2	Buzzard II		1	2	3	15	10	1	4	0.4	0.3	0.2	L	30	A	90	90	1.5	2
3	Cardinal I		2	4	5	12	12	1.5	5	0.5	0.4	0.3	L	90	A	90	90	2.3	3
4	Cardinal II		2	4	5	12	12	1.5	10	0.5	0.4	0.3	L	90	A	90	90	4	3
5	Crow I		4	5	8	10	15	5	10	0.9	0.8	0.7	B	20	D	95	90	5	9
6	Crow II		3	4	7	9	15	6	10	0.8	0.7	0.6	B	30	D	95	90	6	9
7	Pigeon I		5	7	10	8	18	8	10	0.9	0.8	0.7	B	20	D	95	90	6.5	7
8	Pigeon II		4	6	9	7	18	9	10	0.8	0.7	0.6	B	30	D	95	90	7.5	7
9	Robin I		7	12	17	6	22	10	10	0.9	0.8	0.7	B	20	D	98	90	8.3	6
10	Robin II		6	11	16	6	22	11	10	0.8	0.7	0.6	B	30	D	98	90	8.8	6
11	Dove I		10	15	20	4	30	23	10	0.9	0.8	0.7	B	20	D	98	90	9.3	5
12	Dove II		9	14	19	4	30	24	10	0.8	0.7	0.6	B	30	D	98	90	9.9	5

With 204 measurements (12 alternatives scored against 17 measures) taken, some would be tempted to claim success and call it a day, but the lead systems engineer for the sUAV trade-off analysis knew there was much to be done in order to fully mine this data for understanding and to communicate the findings and recommendations to the study sponsor in a way that would lead to action. Of course, making 204 measurements and recording them in a well-structured data store is no small task, and it is better than some of the trade-study products he has seen over his career, but the consequence scorecard alone is certainly not conducive to confident decision-making. The next section of this chapter discusses some of the best practices for synthesizing results for rapid and thorough understanding of the trade at hand followed by an application of these best practices to the sUAV case study.

5.3.5 Synthesize Results

At this point in the process, the decision team has generated a large amount of data as summarized in the consequences scorecard. Now it is time to explore the data and display results in a way that facilitates understanding. Transforming the data in the consequences scorecard into a value scorecard is accomplished through the use of the value functions developed in the decision analysis process step described earlier. Table 5.15 shows the structure of a value scorecard. In an effort to enhance speed and depth of comprehension of the value scorecard, consider associating increments on the value scale with a color according to heat map conventions. This view can be useful when trying to determine which objectives are causing a particular alternative trouble. In addition, one can use this view to quickly see if there are objectives for which no alternative scores well. From this view, the systems engineer can also see if there is at least one alternative that scores above the walk-away point for all objectives. If not, the solution set is empty and the decision team needs to generate additional alternatives or adjust objective measures.

Radar graphs and tornado graphs (Figures 5.12 and 5.13) are popular visualization techniques to show the same value data captured in a Heat-Indexed Value Scorecard discussed earlier but usually for only two alternatives at a time.

5.3.6 Develop Multidimensional Value Model

Beyond the consequence scores for each alternative on each measure, all that was needed to construct the visualizations covered in Table 5.16 were the value functions associated with each objective measure. By introducing the weighting scheme, the systems engineer can create aggregated value visualizations. The first step in assessing an alternative's aggregated value is a prescreen for alternatives that fail to meet a walk-away point for any objective measure and set that alternative's aggregated value to zero regardless of how it performs on other objective measures. For those alternatives that pass the walk-away prescreen, the additive value model² uses the following equation to calculate each alternative's aggregated value:

5.1

where v(x) is the alternative's value, i = 1 to n is the number of the measure, x_i is the alternative's score on the ith measure, v_i(x_i) = is the single-dimensional value of a score of x_i, w_i is the weight of the ith measure,

5.2

and (all weights sum to 1).

Table 5.16 Value Scorecard for sUAV example

This chapter is devoted to the pragmatic application of the aggregation technique but a thorough treatment of the mathematical foundation for the additive value model is provided in Chapter 2 and by (Keeney 1981; Stewart 1996; Von Winterfeldt et al., 1986).

With the weights in hand, one can construct aggregated visualizations such as the value component graph as shown in Figure 5.14. In a value component graph, each alternative's total value is represented by the total length of a segmented bar. Each bar segment represents the contribution of the value earned by the alternative of interest within a given measure by the weighted value (Parnell et al. 2013). As discussed in Section 5.3.2 and illustrated in Table 5.3, every objective has one measure and only one measure.

The heart of a decision support process for systems engineering trade analysis is the ability to integrate otherwise separate analyses into a coherent, system-level view that traces consequences of design decisions across all dimensions of stakeholder value. The stakeholder value scatterplot illustrated in Figure 5.15 shows in one chart how all system-level alternatives respond in multiple dimensions of stakeholder value.

Figure 5.15 illustrates the structure of a stakeholder value scatterplot showing how the six hypothetical alternatives respond to four dimensions of stakeholder value – performance value, life cycle cost, development schedule, and long-term viability. Each system alternative is represented by a scatterplot marker. An alternative's life cycle cost and performance value are indicated by a marker's x and y positions, respectively. An alternative's development duration is indicated by the color of the marker per heat map conventions shown in the legend, while the long-term viability of a particular alternative is indicated by the shape of the marker as described in the legend.

5.3.6.1 Example of Developing a Multidimensional Value Model

Resuming the sUAV case study example, the lead systems engineer is anxious to respond to the study sponsor feedback and provide a more explicit representation of the cost, schedule, and performance trade between the alternatives under consideration. Making use of the additive value model and the weighting scheme described in Figure 5.10 along with the value scorecard in Table 5.15 the lead systems engineer is able to create the aggregated value visualizations shown in Figures 5.16 and 5.17.

The lead systems engineer was excited about the value scatterplot and once again hurried to the show this visualization to the study sponsor and several other stakeholders. The feedback he received was again glowing, agreeing that this particular visualization clearly showed the cost/schedule/performance trade. However, this time the study sponsor asked to understand how variations in priority weightings would impact the results. The next section in this chapter covers the best practices associated with identifying uncertainty and conducting probabilistic analysis followed by an application of these techniques to the sUAV case study.

5.3.7 Identify Uncertainty and Conduct Probabilistic Analysis

As part of the assessment, it is important for the subject matter expert to explicitly discuss potential uncertainty surrounding the assessed score and variables that could impact one or more scores. One source of uncertainty that is common within systems engineering trade-off analyses that explore various system architectures is technology immaturity. System design concepts are generally described as a collection of subsystem design choices, but if some of the design choices include technologies that are immature, there may be a lack of detail associated with component-level design decisions that will eventually be made downstream during detailed design. Many times the subject matter expert can assess an upper, nominal, and lower bound measure response by making three separate assessments (i) assuming low performance, (ii) assuming moderate performance, and (iii) assuming high performance.

Another source of uncertainty has to do with the subjective nature of the value schemes elicited from the stakeholders. Considering that a stakeholder's value scheme is often tied to their forecast of future scenarios and acknowledging that the stakeholder is probably not clairvoyant leads to the conclusion that people can reasonably disagree about things such as priority weightings. One of the common pitfalls of systems engineering trade-off analyses is to collect value scheme information from a small set of like-minded stakeholders and ignore the fact that value schemes likely vary across the full population of stakeholders. The best practice that should be employed here is to collect value scheme information from many different stakeholders and then run a battery of sensitivity analyses pertaining to priority weightings to ensure that a meaningful decision can be made in the presence of such uncertainty. The next section of this chapter applies some of techniques for identifying uncertainty to the sUAV example, and the subsequent section covers techniques used to assess the impact of uncertainty.

5.3.7.1 Example of Identifying Uncertainty and Conducting Probabilistic Analysis

Recall the sUAV case study and how the study sponsor asked the lead systems engineer to show how variations in priority weightings would impact the results. Toward this end, the lead systems engineer discussed the composition of the focus group he used to develop the priority weightings with the study sponsor and asked him to provide some recommendations regarding stakeholder representatives that may have a different view on how weightings for this trade should be set. The study sponsor provided the systems engineering lead with a list of contacts that he suspects would offer a somewhat different take on weights associated with this trade. Focus group number 2 was formed from this list, and the lead systems engineer developed Figure 5.18 to highlight the differences in the priority weightings generated by the two groups.

The lead systems engineer noticed the clear differences between the two group's areas of emphasis, group 1 on ISR data collection quality and group 2 on soldier mobility while transporting the sUAV system. He realized that capturing this source of uncertainty is the first step to assessing its impact on the overall decision. The next section of this chapter describes some sensitivity analyses that can be used to gain this understanding followed by a return the sUAV case study.

5.3.8 Assess Impact of Uncertainty

Decision analysis uses many forms of sensitivity analysis including line diagrams, tornado diagrams, and waterfall diagrams and several uncertainty analyses including Monte Carlo Simulation, decision trees, and influence diagrams (Parnell et al., 2013). Due to space limits, only line diagrams of sensitivity to weighting and Monte Carlo Simulation are discussed in this section.

Many decision-makers will want to understand how sensitive a particular recommendation is to weightings and will ask questions regarding the degree to which a particular weighting would need to be changed in order to change in recommended alternative. A common approach to visualizing the impact of measure weighting on overall value is to sweep each measure's weighting from absolute minimum to absolute maximum while holding the relative relationship between the other measure weightings constant and noting changes to overall score. The output of this type of sensitivity analysis is in the form of a line graph (Parnell et al., 2011). An example of such a graph is provided in Figure 5.19. Note that this particular example shows how sweeping the weight associated with Objective 1.1.2 impacts performance value. The graph in this example shows that the alternative with the highest performance value is alternatives 2 for all cases where priority weighting associated with Objective 1.1.2 is somewhat low but as the weight of Objective 1.1.2 is increased to 0.8 and above, alternative 4 emerges as the high performer.

Sometimes, it is useful to consider all uncertainties at once instead of merely investigating the impact of one particular uncertainty at a time. For this type of view, consider the tradespace visualization in Figure 5.20. Once all the uncertainties have been assessed, Monte Carlo Simulations (Chapter 3) can be executed to identify the uncertainties that impact the decision findings and of the uncertainties that are inconsequential. For example, Figure 5.20 shows that after considering all sources of uncertainty, alternative 1 is less susceptible to changes in stakeholder value than alternative 3. Note, however, that although the stakeholder value of alternative 3 is a bit more volatile in the presence of uncertainty, its stakeholder value never falls below the highest level of alternative 1's stakeholder value. This graph also indicates that alternative 3 has the edge on long-term viability, whereas development duration differentiation is nil. Consequently, decision-makers may be inclined to pursue alternative 3 over alternative 1 if the differences in life cycle costs were deemed affordable.

The takeaway of this section may be that good decisions can often be made in the presence of high uncertainty. Systems engineers should not let what is not known sabotage a decision-making opportunity if what is known is sufficient.

5.3.8.1 Example of Assessing the Impact of Uncertainty

Picking up the sUAV case study, the lead systems engineer applied two sensitivity analysis techniques to assess the impact of the uncertainty surrounding priority weightings. As a first step in his attempt to get his arms around the degree of decision volatility introduced by changes in weighting schemes, he generated a full set of performance value sensitivity line graphs – one graph per measure. Figure 5.21 shows one of the these line graphs, the line graph that shows changes in functional performance value as the priority weight associated with “avoid impeding soldier sprint” objective is swept from 0 to 1. Notice that the top performing alternative only changes twice throughout the entire sweep – from Crow I to Crow II at 0.45 and from Crow II to Buzzard II at 0.75.

Although the set of line graphs were interesting and shed some light on the degree of volatility pertaining to this specific decision, the lead systems engineer feared that such graphs would not directly address the question raised by the study sponsor. For this, he decided to generate a stakeholder value scatterplot with uncertainty. Figure 5.22 shows this scatterplot. Notice how this graph clearly shows that Crow I maintains the highest functional performance value under either weight set considered.

The study sponsor was thrilled when the lead systems engineer presented the visualization of Figure 5.22, and the graph formed the focal point for many thoughtful negotiations among the stakeholders and the decision authority.

5.3.9 Improve Alternatives

One could be tempted to end the decision analysis here, highlight the alternative that has the highest total value, and claim success. Such a premature ending, however, would not be considered best practice. Mining the data generated for the first set of alternatives will likely reveal opportunities to modify some subsystem design choices to claim untapped value and reduce risk. Recall the cyclic decision process map and the implied feedback. Taking advantage of this feedback loop and using initial findings to generate new and creative alternatives starts the process of transforming the decision process from “Alternative-Focused Thinking” to “Value-Focused Thinking” (Keeney 1992). To complete the transformation from alternative-focused thinking to value-focused thinking, consider taking additional steps to spark focused creativity to overcome anchoring biases. As Keeney warns,

Once a few alternatives are stated, they serve to anchor thinking about others. Assumptions implicit in the identified alternatives are accepted, and the generation of new alternatives, if it occurs at all, tends to be limited to a tweaking of the alternatives already identified. Truly creative or different alternatives remain hidden in another part of the mind, unreachable by mere tweaking. Deep and persistent thought is required to jar them into consciousness. (Keeney 1993)

To help generate a creative and comprehensive set of alternatives, consider conducting an alternative generation table (also called a morphological box) (Buede, 2009; Parnell et al., 2011) analysis to generate new alternatives. Chapter 8 presents this and other alternative generation techniques.

5.3.10 Communicating Trade-Offs

This is the point in the process where the decision team identifies key observations regarding what stakeholders seem to want and what they must be willing to give up in order achieve it. It is here where the decision team can highlight the design decisions that most influence shareholder and stakeholder value and that are inconsequential. In addition, the important uncertainties and risks should also be identified. Observations regarding combination effects of various design decisions are also important products of this process step. Competing objectives that are driving the trade should be explicitly highlighted as well

Beyond the top-level tradespace visualization products, the decision analyst must be able to rapidly drill down to supporting rationale. The decision support tool construct represented in Figure 5.23 will allow the decision team to navigate seamlessly from top-level stakeholder value scatterplot all the way down to any structured scoring sheet so that rationale for any given score is only a click away. Rapid access to rationale associated with the derivation of the value function or priority weightings is also essential for full traceability.

5.3.11 Present Recommendation and Implementation Plan

It is often helpful to describe the recommendation in the form of clearly worded, actionable task list to increase the likelihood of the decision process leading to some form of action, thus delivering some tangible value to the sponsor. Reports are important for historical traceability and future decisions. Take the time and effort to create a comprehensive, high-quality report detailing study findings and supporting rationale. Consider static paper reports augmented with dynamic hyperlinked e-reports.