Preface
Hardly anyone gets into software development because they were forced into it. Many literally fall in love with programming and decide to pursue it for a living. And yet there is a vast gap between what most hoped their career would be like and the dark, depressing reality of software development. The software industry as a whole is in a deep crisis. What makes the crisis so acute is that it is multidimensional; every aspect of software development is broken:
Cost. There is weak correlation between the budget set for a project and what it will actually cost to develop the system. Many organizations do not even try to address the cost issue, perhaps because they simply do not know how, or perhaps because doing so will force them to recognize they cannot afford the system. Even if the cost of the first version of a new system is justified, often the cost across the life of the system is much higher than what it should have been due to poor design and an inability to accommodate changes. Over time, maintenance costs become so prohibitive that companies routinely decide to wipe the slate clean, only to end up shortly thereafter with an equally or even more expensive mess as a new system. No other industry opts for a clean slate on a regular basis simply because doing so does not make economic sense. Airlines maintain jumbo jets for decades, and a house can be a century old.
Schedule. Deadlines are often just arbitrary and unenforceable constructs because they have little to do with the time it takes to actually develop the system. For most developers, deadlines are these useless things whooshing by as they plow ahead. If the development team does meet the deadline, everyone is surprised because the expectation is always for them to fail. This, too, is a direct result of a poor system design that causes changes and new work to ripple through the system and invalidate previously completed work. Moreover, it is the result of a very inefficient development process that ignores both the dependencies between activities and the fastest, safest way of building the system. Not only is the time to market for the whole system exceedingly long, but the time for a single feature may be just as inflated. It is bad enough when the project slips its schedule; it is even worse when the slip was hidden from management and customers since no one had any idea what the true status of the project was.
Requirements. Developers often end up solving the wrong problems. There is a perpetual communication failure between the end customers or their internal intermediaries (such as marketing) and the development team. Most developers also fail to accommodate their failure to capture the requirements. Even when requirements are perfectly communicated, they will likely change over time. This change invalidates the design and unravels everything the team tried to build.
Staffing. Even modest software systems are so complex that they have exceeded the capacity of the human brain to make sense of them. The internal and external complexity is a direct result of poor system architecture, which in turn leads to convoluted systems that are very difficult to maintain, extend, or reuse.
Maintenance. Most software systems are not maintained by the same people who developed them. The new staff does not understand how the system operates, and as a result they constantly introduce new problems as they try to solve old ones. This quickly drives up the cost of maintenance and the time to market, and leads to clean-slate efforts or canceled projects.
Quality. Perhaps nothing is as broken with software systems as quality. Software has bugs, and the word “software” is itself synonymous with “bugs.” Developers cannot conceive of defect-free software systems. Fixing defects often increases the defect count, as does adding features, or just plain maintenance. Poor quality is a direct result of a system architecture that does not lend itself to being testable, understandable, or maintainable. Just as important, most projects do not account for essential quality-control activities and fail to allocate enough time for every activity to be completed in an impeccable manner.
Decades ago, the industry started developing software to solve the world’s problems. Today, software development itself is a world-class problem. The problems of software development frequently manifest themselves in nontechnical ways such as a high-stress working environment, high turnover rate, burnout, lack of trust, low self-esteem, and even physical illness.
None of the problems in software development is new.1 Indeed, some people have spent their entire careers in software development without seeing software done right even once. This leads them to believe that it simply cannot be done, and they are dismissive of any attempt to address these issues because “that’s just the way things are.” They may even fight those who are trying to improve software development. They have already concluded that this goal is impossible, so anyone who is trying to get better results is trying to do the impossible, which insults their intellect.
1. Edsger W. Dijkstra, “The Humble Programmer: ACM Turing Lecture,” Communications of the ACM 15, no. 10 (October 1972): 859–866.
My own track record is a counterexample demonstrating that it is possible to successfully develop software systems. Every project for which I was responsible shipped on schedule, on budget, and with zero defects. I continued this record after founding IDesign, where we have helped customers again and again deliver on their commitments.
This consistent, repeatable track record of success was no accident. My training and schooling were in systems engineering, of both physical systems and software systems, and it was easy to recognize the similarities across the two worlds. Applying practical principles to software design, ideas that are common-sense in other engineering fields made sense in software systems, too. It never occurred to me not to treat software development as engineering or to develop a system without a design or without a plan. I saw no need to compromise on my conviction, or to give in to expediencies because doing the right things just worked, and the appalling consequences of not doing so were plain to see. I was fortunate to have great mentors, to be at the right place at the right time to see what worked and what did not, to have the opportunity to participate early on in large critical efforts, and to be part of cultures of excellence.
In recent years, I have noticed that the industry’s problems are getting worse. More and more software projects fail. These failures are getting more expensive in both time and money, and even completed projects tend to stray further afield from their original commitments. The crisis is worsening not just because the systems are getting bigger or because of the cloud, aggressive deadlines, or higher rate of change. I suspect the real reason is that the knowledge of how to design and develop software systems is slowly fading from within the development ranks. Once, most teams had a veteran who mentored the young and handed down the tribal knowledge. Nowadays these mentors have moved on or are retiring. In their absence, the rank and file is left with access to infinite information but zero knowledge.
I wish there was just one thing you could do to fix the software crisis such as using a process, a development methodology, a tool, or a technology. Unfortunately, to fix a multidimensional problem, you need a multidimensional solution. In this book I offer a unified remedy: righting software.
In the abstract, all I suggest is to design and develop software systems using engineering principles. The good news is that there is no need to reinvent the wheel. Other engineering disciplines are quite successful, so the software industry can borrow their key universal design ideas and adapt them to software. You will see in this book a set of first principles in software engineering, as well as a comprehensive set of tools and techniques that apply to software systems and projects. To succeed, you have to assume an engineering perspective. Ensuring that the software system is maintainable, extensible, reusable, affordable, and feasible in terms of time and risk are all engineering aspects, not technical aspects. These engineering aspects are traced directly to the design of the system and the project. Since the term software engineer largely refers to a software developer, the term software architect has emerged to describe the person in the team who owns all the design aspects of the project. Accordingly, I refer to the reader as a software architect.
The ideas in this book are not the only things you will need to get right, but they certainly are a good start because they treat the root cause of the problems mentioned earlier. That root cause is poor design, be it of the software system itself or of the project used to build that system. You will see that it is quite possible to deliver software on schedule and on budget and to design systems that meet every conceivable requirement. The results are also systems that are easy to maintain, extend, and reuse. I hope that by practicing these ideas you will right not just your system but your career and rekindle your passion for software development.
How This Book Is Organized
The book demonstrates a structured engineering approach to system and project design. The methodology has two parts, reflected by the structure of this book: system design (commonly known as architecture) and project design. Both parts complement each other and are required for success. The appendices provide some supplemental content to the main discussion.
In most technical books, each chapter addresses a single topic and discusses it in depth. This makes the book easier to write, but that is typically not how people learn. In contrast, in this book, the teaching is analogous to a spiral. In both parts of the book, each chapter reiterates ideas from the previous chapters, going deeper or developing ideas using additional insight across multiple aspects. This mimics the natural learning process. Each chapter relies on those that preceded it, so you should read the chapters in order. Both parts of the book include a detailed case study that demonstrates the ideas as well as additional aspects. At the same time, to keep the iterations concise, as a general rule I usually avoid repeating myself, so even key points are discussed once.
Here is a brief summary of the chapters and appendices:
Chapter 1, The Method
Chapter 1 introduces this key idea: To succeed, you must design both the system and the project to build it. Both designs are essential for eventual success. You cannot design the project without the architecture, and it is pointless to design a system that you cannot build.
Part 1: System Design
Chapter 2, Decomposition
Chapter 2 is dedicated to decomposing the system into the components that make up its architecture. Most people decompose systems in the worst possible way, so the chapter starts with explaining what not to do. Once that is established, you will see how to correctly decompose the system, and learn a set of simple analysis tools and observations that help in that process.
Chapter 3, Structure
Chapter 3 improves on the ideas of Chapter 2 by introducing structure. You will see how to capture requirements, how to layer your architecture, the taxonomy of the components of the architecture, their interrelationships, specific classification guidelines, and some related issues such as subsystems design.
Chapter 4, Composition
Chapter 4 shows how to assemble the system components into a valid composition that addresses the requirements. This short chapter contains several of the book’s key design principles, and it leverages the previous two chapters into a powerful mental tool you will use in every system.
Chapter 5, System Design Example
Chapter 5 is an extensive case study that demonstrates the system design ideas discussed so far. This final iteration of the system design spiral presents an actual system, aligns the system design with the business, and shows how to produce the architecture and validate it.
Part II: Project Design
Chapter 6, Motivation
Since most people have never even heard of—let alone practiced—project design, this chapter introduces the concept and provides the motivation for engaging in project design. This is iteration zero of the project design spiral.
Chapter 7, Project Design Overview
Chapter 7 provides a broad overview of how to design a project. It starts by defining success in software development, and then presents the key concepts of educated decisions, project staffing, project network, critical path, scheduling, and cost. The chapter covers most of the ideas and techniques used in subsequent chapters, and it ends with an important discussion of roles and responsibilities.
Chapter 8, Network and Float
Chapter 8 dives into the project network and its use as a design tool. You will see how to model the project as a network diagram, learn the key concept of float, understand how to use floats in staffing and scheduling, and recognize how floats relate to risk.
Chapter 9, Time and Cost
Chapter 9 defines the possible tradeoffs between time and cost in any project and prescribes ways to accelerate any project by working cleaning and correctly. Beyond that, you will learn the key concepts of compression, the time–cost curve, and the elements of cost.
Chapter 10, Risk
Chapter 10 presents the missing element in most projects: quantified risk. You will see how to measure and map risk to the time and cost concepts from the previous chapter, and how to calculate risk based on the network. Risk is often the best way of evaluating options and is a first-class planning tool.
Chapter 11, Project Design in Action
Chapter 11 puts all the concepts of the previous chapters into use via a systematic walkthrough of the steps involved in designing a project. While it has the makings of an example, the objective is to demonstrate the thought process used when designing a project, as well as how to prepare for review by business decision makers.
Chapter 12, Advanced Techniques
Following the spiral model of learning, this chapter offers advanced techniques and concepts. These techniques are useful in projects with all levels of complexity, from the simple to the most challenging. These advanced techniques complement the previous chapters and each other, and you will often use them in combination.
Chapter 13, Project Design Example
Chapter 13 is the project design example corresponding to the system design example of Chapter 5. It, too, is a case study demonstrating the end-to-end process of designing a project. The focus in this chapter is on the case study and less about the techniques.
Chapter 14, Concluding Thoughts
This final chapter takes a step back from the technical aspects of design and offers a collection of guidelines, tips, perspectives, and development process ideas. It starts by answering the important question of when to design a project, and it ends with the effect project design has on quality.
Appendices
Appendix A, Project Tracking
Appendix A shows you how to track the project’s progress with regard to the plan and how to take corrective actions when needed. Project tracking is more about project management than it is about project design, but it is crucial in assuring you meet your commitments once the work starts.
Appendix B, Service Contract Design
The architecture itself is broad and coarse, and you have to design the details of each of its components. The most important of these details is the service contract. Appendix B points you toward the correct way of designing service contracts. In addition, the discussion of modularity, size, and cost resonates very well with most chapters in this book.
Appendix C, Design Standard
Appendix C is a consolidated list of the key directives, guidelines, and dos and don’ts mentioned throughout this book. The standard is terse and is all about the “what,” not the “why.” The rationale behind the standard is found in the rest of the book.
Some Assumptions About the Reader
While this book targets software architects, it has a much broader audience. I assume that you, the reader, are an architect or senior software professional, a project manager, or someone who wears multiple hats. That said, aspiring developers wanting to grow their skill set will benefit greatly from the book. Regardless of your current position, this book will open doors for you through the rest of your career. You may not be an accomplished architect when you first pick up this book, but you will be among the top in the world once you have read it and have mastered the methodology.
The techniques and ideas for the book apply regardless of programming language (such as C++, Java, C#, and Python), platform (such as Windows, Linux, mobile, on-premise, and cloud), and project size (from the smallest to the largest projects). They also cross all industries (from healthcare to defense), all business models, and company sizes (from the startup to the large corporation).
The most important assumption I have made about the reader is that you care about what you do, at a deep level, and the current failures and waste distresses you. You want to do better but lack guidance or are confused by bad practices.
What You Need to Use This Book
The only prerequisite for this book is an open mind. Past failures and frustration are a plus.
Conventions Used in This Book
The book uses the following typographic conventions:
Bold
Used for defining terms and concepts.
Directive
Used for first principles, design rules, or key guidance and advice.
Reserved Words
Used for when referring to reserved words of the methodology.
Additional Online Resources
The web page for this book provides sample files, addenda, and errata. You can access this page at the following address:
http://www.rightingsoftware.org
You will find the example files and related supporting material in this book under the “Download Support Files” link.
For additional information about this book, go to
informit.com/title/9780136524038.
You can also contact the author at this address:
Register Your Product
Register your copy of Righting Software on the InformIT site for convenient access to updates and/or corrections as they become available. To start the registration process, go to informit.com/register and log in or create an account. Enter the product ISBN (9780136524038) and click Submit. Look on the Registered Products tab for an Access Bonus Content link next to this product, and follow that link to access any available bonus materials. If you would like to be notified of exclusive offers on new editions and updates, please check the box to receive email from us.
Acknowledgments
Let me start by thanking the two who urged me to write the book, each in their own unique way: Gad Meir and Jarkko Kemppainen.
Thanks go to the development editor and sounding board, Dave Killian: Any more editing and I would have to list you as a co-author. Next, thanks to Beth Siron for reviewing the raw manuscript. The following people contributed their time by reviewing the draft: Chad Michel, Doug Durham, George Stevens, Josh Loyd, Riccardo Bennett-Lovsey, and Steve Land.
Finally, I am grateful to my wife, Dana, who keeps inspiring me to write and makes it possible for me to take the time away from the family; and to my parents, who imparted to me the love for engineering.