8. Analysis in the Design Process
While this is a book focused on architectural design, we have always believed that design and analysis are two sides of the same coin. Design is the process of making decisions; analysis is the process of understanding those decisions, so that the design may be evaluated. To reflect this intimate relationship, we now turn our attention to why, when, and how to analyze architectural decisions during the design process. We look at various techniques for analysis, discuss when they can be done, and explore their costs and benefits.
8.1 Analysis and Design
Analysis is the process of breaking a complex entity into its constituent parts as a means of understanding it. The opposite of analysis is synthesis. Analysis and design are therefore intertwined activities. During the design process, the activity of analysis can refer to several aspects:
Studying the inputs to the design process to understand the problem whose solution you are about to design. This includes giving priority to the drivers as discussed in Section 3.2.2. This type of analysis is performed in steps 1 and 2 of ADD.
Studying the alternative design concepts that you identified to solve a design problem so as to select the most appropriate one. In this situation, analysis forces you to provide concrete evidence for your choices. This activity is performed in step 4 of ADD and was discussed in Section 3.2.4.
Ensuring the decisions made during the design process (or an iteration) are appropriate. This is the type of analysis that you perform in step 7 of ADD.
The decisions that you make when designing the architecture are not only critical to achieve the quality attribute responses, but frequently the cost associated with correcting them at a later time can be significant, as these decisions may affect many parts of the system. For these reasons, it is necessary to perform analysis during the design process, so that problems can be identified, possibly quantified, and corrected quickly. Remember, being too confident and following your gut instincts may not be the best idea (see the sidebar “‘I believe’ Isn’t Good Enough”). Fortunately, if you have followed the recommendations that we have given up to this point, you should be able to conduct analysis either by yourself or with the help of peers by using the preliminary sketches and views that have been produced as you perform the design process.
Even if you are following a systematic approach to designing your architecture and using design concepts from well-established sources, and even if you have nice-looking diagrams that represent your structures, nothing really guarantees that the decisions you are making will actually satisfy a particular quality attribute scenario. Certain quality attributes are critical to the success of your system; particularly for these decisions, the rationale of “I believe” is not good enough. Studies of practicing software architects have shown that most follow an “adequacy” approach to making design decisions—that is, they adopt the first decision that appears to meet their needs. All too often, they have no rationale to substantiate those decisions other than their gut instincts, their beliefs, based on their (inevitably limited) experience. Thus important decisions are frequently made after insufficient reasoning, which can add risk to a system.
For drivers that are critical to your system, you owe it to yourself and to your organization to perform a more detailed analysis rather than just trusting your gut instinct, relying on analogy and history, or performing a couple of superficial tests to ensure that your drivers are satisfied. The following options will deepen your analysis and hence support your rationale for the decisions made:
Analytic models. These well-established mathematical models allow you to study quality attributes such as performance or availability. They include Markov and statistical models for availability, and queuing and real-time scheduling theory for performance. Analytic models—particularly those that address performance—are highly mature but may require considerable education and training to be used adequately.
Checklists. Checklists are useful tools that allow you to ensure in a systematic way that certain decisions that need to be taken into account are not forgotten. Checklists are available for particular quality attributes in the public domain—for example, the OWASP checklist guides you in performing black box security testing of web applications. Also, your organization may develop proprietary checklists that are specific to the application domains that you are developing. Tactics-based questionnaires, which we will discuss shortly, are a type of checklist for the most important quality attributes, based on the use of tactics.
Thought experiments, reflective questions, and back-of-the-envelope analyses. Thought experiments are informal analyses performed by a small group of designers in which important scenarios are studied to identify potential problems. For example, you might use a sequence diagram produced inside step 5 of ADD and perform a walk-through of the interaction of the objects that support the scenario modeled in the diagram with a colleague. Reflective questions (discussed in depth in Section 8.5) are questions that challenge the assumptions included in the decision-making process. Back-of-the-envelope analyses are rough calculations that are less precise than analytic models, but can be performed quickly. These calculations, which are frequently based on analogies to other similar systems or on prior experience, are useful to obtain ballpark estimates for desired quality attribute responses. For example, by summing the latencies of a number of processes in a pipeline, you can derive a crude estimate of the end-to-end latency.
Prototypes, simulations, and experiments. Purely conceptual techniques for analyzing a design are sometimes inadequate to accurately understand whether certain design decisions are appropriate, or whether you should favor one particular technology over another. In such situations, the creation of prototypes, simulations, or experiments can be an invaluable option to obtain a better understanding. For example, in the back-of-the-envelope estimate of latency described previously, you may not have taken into account that several of the processes are sharing (and hence competing for) the same resources; thus we cannot simply sum their individual latencies and expect to get accurate results. Prototypes and simulations provide a deeper understanding of system dynamics, but may require a significant effort that needs to be considered in the project plan.
As always, none of these techniques is inherently better than the others. Thought experiments and back-of-the-envelope calculations are inexpensive and can be done early in the design process, but their validity may be questionable. Prototypes, simulations, and experiments typically produce much higher-fidelity results, but at a far greater cost. The choice of which technique to employ depends on the context, the risk involved, and the priorities of your quality attributes.
Even so, applying any of these techniques will be helpful in going from “I believe” (that my design is appropriate) to an approach that is backed by documented evidence and argumentation.
8.2 Why Analyze?
Analysis and design are two sides of the same coin. Design is (the process of) making decisions. Analysis is (the process of) understanding the consequences—in terms of cost, schedule, and quality—of those decisions. No sensible architect would make any decision, or at least any nontrivial decision, without first attempting to understand the implications of that decision: its near-term effects and possibly its long-term consequences. Architects, of course, make thousands of decisions in the course of designing a large project, and clearly not all of them matter. Furthermore, not all of the decisions that matter are carriers of quality attributes. Some may deal with which vendor to select, or which coding convention to follow, or which programmer to hire or fire, or which IDE to use—important decisions, to be sure, but not ones that are directly linked to a quality attribute outcome.
Of course, some of these decisions will affect the achievement of quality attributes. When the architect breaks down the development into a system of layers or modules, or both, this decision will affect how a change ripples through the code base, who needs to talk to who when adding a feature or fixing a bug, how easy or difficult it is to distribute or outsource some of the development, how easy it is to port the software to a different platform, and so forth. When the architect chooses a distributed resource management system, how it determines which services are masters and which are slaves, how it detects failures, and how it detects resource starvation will all affect the availability of the system.
So when and why do we analyze during the design process? First, we analyze because we can. An architecture specification, whether it is just a whiteboard sketch or something that has been more formally documented and circulated, is the first artifact supporting an analysis that sheds insight into quality attributes. Yes, we can analyze requirements, but we mainly analyze them for consistency and completeness. Until we translate those requirements into structures resulting from design decisions, we will have little to say about the actual consequences of those decisions, their costs and benefits, and the tradeoffs among them.
Second, and more to the point, we analyze because it is a prudent way of informing decisions and managing risk. No design is completely without risk, but we want to ensure that the risks that we take on are commensurate with our stakeholders’ expectations and tolerances. For a banking application or a military application, our stakeholders will demand low levels of risk, and they should be willing to pay accordingly for higher levels of assurance. For a startup company, where time to market is of the essence and budgets are tight, we might be prepared to accept far higher levels of risk. As with every important decision in software engineering, the answer is clear: It depends.
Finally, analysis is the key to evaluation. Evaluation is the process of determining the value of something. Companies are evaluated to determine their share price. A company’s employees are evaluated annually to determine their raises. In each case, the evaluation is built upon an analysis of the properties of the company or employee.
8.3 Analysis Techniques
Different projects will demand different responses to risk. Fortunately we, as architects, have a wide variety of tools at our disposal to analyze architectures. With a bit of planning, we can match our risk tolerance with a set of analysis techniques that both meet our budget and schedule constraints and provide reasonable levels of assurance. The point here is that analysis does not need to be costly or complex. Just asking thoughtful questions is a form of analysis, and that exercise is pretty inexpensive. Building a simple prototype is more expensive, but in the context of a large project this analysis technique may be well worth the additional expense owing to how it explores and mitigates risks, as we saw in Chapter 5.
Examples of (relatively economical, relatively low ceremony) analysis techniques already in widespread use include design reviews and scenario-based analyses, code reviews, pair programming, and Scrum retrospective meetings. Other commonly used, albeit somewhat more costly, analysis techniques include prototypes (throw-away or evolutionary) and simulations.
At the high end of expense and complexity, we can build formal models of our systems and analyze them for properties such as latency or security or safety. When a candidate implementation or a fielded system finally exists, we can perform experiments, including instrumenting running systems and collecting data, ideally from executions of the system that reflect realistic usages.
As indicated in Table 8.1, the cost of these techniques typically increases as you proceed through the software development life cycle. A prototype or experiment is more expensive than a checklist, which is more expensive than an experience-based analogy. This expected cost correlates fairly strongly with the confidence that you can have in the analysis results. Unfortunately, there is no free lunch!
8.4 Tactics-Based Analysis
Architectural tactics (discussed in Section 2.5.4) have been presented thus far as design primitives. However, because these taxonomies are intended to cover the entire space of architectural design possibilities for managing a quality attribute, we can use them in an analysis setting as well. Specifically, we can use them as guides for interviews or questionnaires. These interviews help you, as an analyst, to gain rapid insight into the architectural approaches taken or not taken.
Consider, for example, the tactics for availability, shown in Figure 8.1.
Each of these tactics is a design option for the architect who wants to design a highly available system. Used in hindsight, however, they represent a taxonomy of the entire design space for availability and hence can be a way of gaining insight into the decisions made, and not made, by the architect. To do this, we simply turn each tactic into an interview question. For example, consider the (partial) set of tactics-inspired availability questions in Table 8.2.
When the questions in Table 8.2 are used in an interview setting, we can record whether each tactic is supported by the system’s architecture, according to the opinions of the architect. For example, in the table, the questions have been answered with respect to design decisions made for the FCAPS system presented in Chapter 4. Note that the answers shown in the table are rather succinct because this is an example; more detailed explanations are encouraged in real-world applications. If we are analyzing an existing system we can additionally investigate the following issues:
Whether there are any obvious risks in the use (or nonuse) of this tactic. If the tactic has been used, we can record here how it is realized in the system (e.g., via custom code, frameworks, or other externally produced components). For example, we might note that the active redundancy tactic has been employed by replicating the application server and other critical components such as the database (as in the case study presented in Chapter 4).
The specific design decisions made to realize the tactic and where in the code base the implementation (realization) may be found. This information is useful for auditing and architecture reconstruction purposes. Continuing the example from the previous bullet, we might probe how many replicas of the application server have been created and where these replicas are located (e.g., on the same rack in a data center, on different racks, in different data centers).
Any rationale or assumptions made in the realization of this tactic. For example, we might assume that there will be no common-mode failure, so it is acceptable that the replicas are identical virtual machines, running on identical hardware.
While this interview-based approach might sound simplistic, it can actually be quite powerful and insightful. In your daily activities as an architect, you may not always take the time to step back and consider the bigger picture. A set of interview questions such as those shown in Table 8.2 force you to do just that. This approach is also quite efficient: A typical interview for a single quality attribute takes between 30 and 90 minutes.
A set of tactics-based questionnaires, covering the seven most important system quality attributes—availability, interoperability, modifiability, performance, security, testability, and usability—can be found in Appendix B. In addition, we have included an eighth questionnaire, on DevOps, as an example of how you can combine the other (more fundamental) questionnaires to create a new questionnaire to address a new set of quality concerns.
8.5 Reflective Questions
Similar to the tactics-based interviews, a number of researchers have advocated the practice of asking (and answering) reflective questions to augment the design process. The idea behind this process is that we actually think differently when we are problem-solving and when we are reflecting. For this reason, researchers have advocated a separate “reflection” activity in design that both challenges the decisions made and challenges us to examine our biases.
Architects, like all humans, are subject to bias. For example, we are subject to confirmation bias—the tendency to interpret new information in a way that confirms our preconceptions—and we are subject to anchoring bias—the tendency to rely too heavily on the first piece of information that we receive when investigating a problem, using this information to filter and judge any subsequent information. Reflective questions help to uncover such biases in a systematic way, which can lead us to revise our assumptions and hence our designs.
In their research on reflective questions, Razavian et al. have proposed that one can and should reflect on context and requirements (Are the contexts and requirements identified relevant, complete, and accurate?), design problems (Have they been properly and fully articulated?), design solutions (Are they appropriate given the requirements?), and design decisions (Are they principled and justified?). Examples of reflective questions that they propose include the following:
Which assumptions are made? Do the assumptions affect the design problem? Do the assumptions affect the solution option? Is an assumption acceptable in a decision?
What are the risks that certain events would happen? How do the risks cause design problems? How do the risks affect the viability of a solution? Is the risk of a decision acceptable? What can be done to mitigate the risks?
What are the constraints imposed by the contexts? How do the constraints cause design problems? How do the constraints limit the solution options? Can any constraints be relaxed when making a decision?
What are the contexts and the requirements of this system? What does this context mean? What are the design problems? Which are the important problems that need to be solved? What does this problem mean? Which potential solutions exist for this problem? Are there other problems to follow up in this decision?
Which contexts can be compromised? Can a problem be framed differently? What are the solution options? Can a solution option be compromised? Are the pros and cons of each solution treated fairly? What is an optimal solution after tradeoff?
Of course, you might not employ all of these questions, and you would not employ this technique for every decision that you make. Used judiciously, however, these kinds of questions can help you to reflect mindfully on the decisions that you are making.
8.6 Scenario-Based Design Reviews
Comprehensive scenario-based design reviews, such as the ATAM, have typically been conducted outside the design process. The ATAM is an example of a comprehensive architecture evaluation (see the sidebar “The ATAM”).
An ATAM review, as it was initially conceived, was a “milestone” review. When an architect or other key stakeholder believed that there was enough of an architecture or architecture description to analyze, an ATAM meeting could be convened. This might occur when an architectural design had been done but before much, if any, implementation had been completed. More commonly, it occurred when an existing system was in place and some stakeholders wanted an objective evaluation of the risks of the architecture before committing to it, evolving it, acquiring it, and so forth.
The ATAM—Architecture Tradeoff Analysis Method (ATAM) is an established method for analyzing architectures, driven by scenarios. Its purpose is to assess the consequences of architectural decisions in light of quality attribute requirements and business goals.
The ATAM brings together three groups in an evaluation:
A trained evaluation team
An architecture’s “decision makers”
Representatives of the architecture’s stakeholders
The ATAM helps stakeholders ask the right questions to discover potentially problematic architectural decisions—that is, risks. These discovered risks can then be made the focus of mitigation activities such as further design, further analysis, prototyping, and implementation. In addition, design tradeoffs are often identified—hence the name of the method. The purpose of the ATAM is not to provide precise analyses: This method typically is applied in two 2-day meetings and this (relatively) short time frame does not permit a deep dive into any specific concern. Those kinds of analyses are, however, appropriate as part of the risk mitigation activities that could follow and be guided by an ATAM.
The ATAM can be used throughout the software development life cycle. For example, it can be used in the following circumstances:
After an architecture has been specified but there is little or no code
To evaluate potential architectural alternatives
To evaluate the architecture of an existing system
The outputs of the ATAM evaluation are as follows:
A concise presentation of the architecture. The architecture is presented in one hour.
A concise articulation of the business goals for the system under scrutiny. Frequently, the business goals presented in the ATAM are being seen by some of the assembled participants for the first time and these are captured in the outputs.
A set of prioritized quality attribute requirements, expressed as scenarios.
A mapping of architectural decisions to quality requirements. For each quality attribute scenario examined, the architectural decisions that help to achieve it are identified and recorded.
A set of sensitivity and tradeoff points. These architectural decisions have a marked effect on one or more quality attributes.
A set of risks and non-risks. A risk is defined as an architectural decision that may lead to undesirable consequences in light of quality attribute requirements. A non-risk is an architectural decision that, upon analysis, is deemed safe. The identified risks form the basis of an architectural risk mitigation plan.
A set of risk themes. The evaluation team examines the full set of discovered risks to identify overarching themes that reveal systemic weaknesses in the architecture (or perhaps even in the architecture process and team). If left untreated, these weaknesses will threaten the project’s ability to meet the business goals.
There are also intangible results of an ATAM-based evaluation: a sense of community developed among the stakeholders, open communication channels between the architect and the stakeholders, a better overall understanding of the architecture and its strengths and weaknesses. While these results are difficult to measure, they are no less important than the others and often are the longest-lasting artifacts.
An ATAM evaluation takes place in four phases. The first phase (phase 0) and the final phase (phase 3) are managerial: setting up the evaluation at the start and reporting results and follow-on activities at the end. The middle phases (phases 1 and 2) are when the actual analysis takes place. The steps enacted in phases 1 and 2 are as follows:
1. Present the ATAM
2. Present the business drivers
3. Present the architecture
4. Identify the architectural approaches
5. Generate a quality attribute utility tree
6. Analyze the architectural approaches
7. Brainstorm and prioritize scenarios
8. Analyze the architectural approaches
9. Present the results
In phase 1, we enact steps 1–6 with a small, internal group of stakeholders—typically just the architect, project manager and perhaps one or two senior developers. In phase 2, we invite a larger group of stakeholders to attend—all the people who attended phase 1 plus external stakeholders, such as customer representatives, end-user representatives, quality assurance, operations, and so forth. In phase 2, we review steps 1–6 and enact steps 7–9.
The actual analysis takes place in step 6, where we analyze architectural approaches by asking the architect to map the highest-priority scenarios, one at a time, onto the architectural approaches that have been described. During this step, the analysts ask probing questions, motivated by a knowledge of quality attributes, and risks are discovered and documented.
The idea of having a separate, distinct evaluation activity once the architecture is “done” fits poorly with the way that most organizations operate today. Today, most software organizations are practicing some form of Agile or iterative development. There is no distinct monolithic “architecture phase” in Agile processes. Rather, architecture and development are co-created in a series of sprints. For example, as discussed in Chapter 2, many Agile thought leaders are promoting practices such as “disciplined agility at scale”, the “walking skeleton”, and the “scaled Agile framework”, all of which embrace the idea that architectures continuously evolve in relatively small increments, addressing the most critical risks. This may be aided by developing a small proof-of-concept or minimum viable product (MVP), or doing strategic prototyping.
To better align with this view of software development, a lightweight scenario-based peer review method, based on the ATAM, has been promoted. A lightweight ATAM evaluation can be conducted in a half-day meeting. It can also be carried out internally, using just project members. Of course, an external review gives more objectivity and may produce better results, but this exercise may be too costly or infeasible due to schedule or intellectual property (IP) constraints. A lightweight ATAM therefore provides a reasonable middle ground between a costly but more objective and comprehensive ATAM and doing no analysis whatsoever, or only doing ad hoc analysis.
An example schedule for a lightweight ATAM evaluation conducted by project members on their own project is given in Table 8.3.
A half-day review such as this is similar, in terms of effort, to other quality assurance efforts that are typically conducted in a development project, such as code reviews, inspections, and walk-throughs. For this reason, it is easy to schedule a lightweight ATAM evaluation in a sprint, particularly in those sprints where architectural decisions are being made, challenged, or changed.
8.7 Architecture Description Languages
If the application that you are building has stringent quality requirements in the areas of runtime performance (latency, throughput), reliability/availability, safety, or security, then you might consider documenting your design decisions, in the form of architectural structures, in an architecture description language (ADL). ADLs lend themselves to formal, automated analysis, which is precisely why we include them here. ADLs typically employ both a graphical and a (formally defined) textual notation to describe an architecture—primarily the computational (runtime) components and interactions among them—and its properties. The Unified Modeling Language (UML) is the most widely used notation for documenting architectures in industrial practice, though even it is not universally used. Few industrial projects endeavor to describe all, or even most, of their architectures in any ADL.
Some ADLs, such as AADL, strive to be formal models that have precise and decidable semantics. This regimentation means that they can be automatically checked for properties of interest, typically performance, availability, and safety, although in principle other quality attributes can be accommodated. While there is an often a steep learning curve for becoming proficient with the language and the surrounding tool suite, using a formalized ADL offers several benefits. First, an ADL forces you to document your architectural decisions, and hence to explicitly acknowledge when and where your architectural understanding is incomplete or vague. This benefit accrues with any form of documentation—it forces you to be explicit—but is especially true of ADLs. This leads to the second benefit of ADLs: They are typically accompanied by a tool suite that can analyze the architecture description for various properties at the click of a button.
So why are ADLs seldom used outside of academia? A number of possible reasons for this reluctance exist. First, it is not in our common practice. ADLs—even the UML—are typically not taught in computer science or software engineering curricula and are not well supported in most popular IDEs. Second, ADLs are perceived as being challenging to use and not user-friendly, requiring both a large up-front effort and a large continuing effort to maintain. This point is, perhaps, the most significant one: Architects and programmers generally do not want to maintain a second, parallel base of knowledge about their systems. For some systems, this may be the right choice. For others—typically those with stringent and uncompromising quality attribute requirements—having a separate and separately analyzable representation of the design might be the most prudent course of action. In civil engineering, by way of contrast, no project may be approved for construction without first being represented in a separate analyzable document.
8.8 Summary
No one would consider fielding code that they had not tested—yet architects and programmers regularly commit to (implement) architectural decisions that have not been analyzed. Why the dichotomy? Surely, if testing code is important, then “testing” the design decisions you have made is an order of magnitude more important, as these decisions often have long-term, system-wide, and significant impacts.
The most important message of this chapter is that design and analysis are not really separate activities. Every important design decision that you make should be analyzed. A variety of techniques can be applied to do this continuously, in a relatively disruption-free manner, as part of the process of designing and evolving a system.
The interesting questions are not whether to analyze, but rather how much to analyze and when. Analysis is inherent in doing good design, and it should be a continuous process.
8.9 Further Reading
The sets of architectural tactics used here have been documented in L. Bass, P. Clements, and R. Kazman, Software Architecture in Practice (3rd ed.), Addison-Wesley, 2012. The availability tactics were first created in J. Scott and R. Kazman, “Realizing and Refining Architectural Tactics: Availability”, CMU/SEI-2009-TR-006, 2009.
The idea of reflective questions was first introduced in M. Razavian, A. Tang, R. Capilla, and P. Lago, “In Two Minds: How Reflections Influence Software Architecture Design Thinking”, VU University Amsterdam, Tech. Rep. 2015-001, April 2015. The idea that software designers satisfice—that is, they look for a “good enough”, as opposed to an optimal, solution—has been discussed in A. Tang and H. van Vliet, “Software Designers Satisfice”, European Conference on Software Architecture (ECSA 2015), 2015.
The ATAM was comprehensively described in P. Clements, R. Kazman, and M. Klein, Evaluating Software Architectures: Methods and Case Studies, Addison-Wesley, 2001. The lightweight ATAM was first presented in L. Bass, P. Clements, and R. Kazman, Software Architecture in Practice (3rd ed.), Addison-Wesley, 2012. In addition, ATAM-style peer reviews have been described in F. Bachmann, “Give the Stakeholders What They Want: Design Peer Reviews the ATAM Style”, Crosstalk, November/December 2011.
Architecture description languages have a history almost as long as the history of software architecture itself. The most widely used ADL in practice is AADL (Architecture Analysis and Design Language), which is described in P. Feiler and D. Gluch, Model-Based Engineering with AADL: An Introduction to the SAE Architecture Analysis & Design Language, Addison-Wesley, 2013. An overview of, and analysis of industrial requirements for, ADLs can be found in I. Malavolta, P. Lago, H. Muccini, P. Pelliccione, and A. Tang, “What Industry Needs from Architectural Languages: A Survey”, IEEE Transactions on Software Engineering, 39(6):869–891, June 2013.