Praise for Lean-Agile Software Development
“This book is a timely addition to our Agile body of knowledge. Very little has been said to date about how we scale Agile software projects beyond the single team. The authors do an excellent job of explaining the foundations of Lean thinking and how these concepts can be applied across the enterprise. Lean is the key to scaling Agile projects, and this book provides the foundational knowledge you need to make it happen.”
—Mike Cottmeyer, product consultant and
Agile evangelist, VersionOne
“The book brings a pragmatic approach to the difficult transition from early adoption of Agile practices to enabling product development. It is thought provoking in the context of the teams I am currently coaching, and it highlights a number of areas for improvement. I would recommend this book to anyone who is coaching an enterprise-wide Lean-Agile transformation.”
—Kay Johnson, PMP, Agile development consultant
and project manager
“The ideas from the Toyota Production System and Lean manufacturing in general are gradually making their way into the world of software development, and this book provides both a gentle introduction to those unfamiliar with Agile/Lean as well as more advanced material for those who are already practitioners in this area. Worth reading.”
—Mark Needham, application developer,
ThoughtWorks
“For a good few years, when asking why Agile approaches work, we got the response ‘It’s empirical. We tried things and kept the ones that worked.’ Now people have applied theory from the Lean body of knowledge, and it tells us why Agile approaches work. Using this theory, we can make well-reasoned choices about what changes to our ways of working would be improvements, overall. This book is about this synergy between Lean and Agile. For those who believe in magic, find an empirical guru to believe. For the rationalists among us, here’s a good book for you.”
—Paul Oldfield, Capgemini
“This book is a worthy roadmap to a successful adoption of Lean-Agile development and management. You can see in every detail the authors have on-the-job experience. The way they write shows their enthusiasm for Lean, and this motivates the reader to follow the principles and practices in the book. I had a great time reading this book and I am using it daily as a reference.”
—Domingo Chabalgoit, independent IT consultant
“There are many sources of information for Agile software development and Lean systems thinking. Until now, attempts to merge the two have often resulted in frustration, delays, quality issues, and budget overruns. Lean-Agile Software Development is the roadmap to achieving success using Lean-Agile techniques and applying them throughout the enterprise and product-development life cycle.”
—Bob Bogetti, lead system designer,
Baxter Healthcare
Lean-Agile Software Development
Achieving Enterprise Agility
Upper Saddle River, NJ • Boston • Indianapolis • San Francisco
New York • Toronto • Montreal • London • Munich • Paris • Madrid
Capetown • Sydney • Tokyo • Singapore • Mexico City
Illustrations by Andrea Chartier Bain
Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in this book, and the publisher was aware of a trademark claim, the designations have been printed with initial capital letters or in all capitals.
The authors and publisher have taken care in the preparation of this book, but make no expressed or implied warranty of any kind and assume no responsibility for errors or omissions. No liability is assumed for incidental or consequential damages in connection with or arising out of the use of the information or programs contained herein.
The publisher offers excellent discounts on this book when ordered in quantity for bulk purchases or special sales, which may include electronic versions and/or custom covers and content particular to your business, training goals, marketing focus, and branding interests. For more information, please contact:
U.S. Corporate and Government Sales
(800) 382-3419
[email protected]
For sales outside the United States please contact:
International Sales
[email protected]
Visit us on the Web: informit.com/netobjectives
Library of Congress Cataloging-in-Publication Data
Shalloway, Alan.
Lean-agile software development : achieving enterprise agility / Alan Shalloway, Guy Beaver, James
R. Trott.
p. cm.
Includes bibliographical references and index.
ISBN 0-321-53289-9 (pbk. : alk. paper) 1. Agile software development. I. Beaver, Guy. II. Trott,
James. III. Title.
QA76.76.D47S47 2009
005.1—dc22
2009032621
Copyright © 2010 Pearson Education, Inc.
All rights reserved. Printed in the United States of America. This publication is protected by copyright, and permission must be obtained from the publisher prior to any prohibited reproduction, storage in a retrieval system, or transmission in any form or by any means, electronic, mechanical, photocopying, recording, or likewise. For information regarding permissions, write to:
Pearson Education, Inc
Rights and Contracts Department
501 Boylston Street, Suite 900
Boston, MA 02116
Fax: (617) 671-3447
ISBN-13: 978-0-321-53289-3
ISBN-10: 0-321-53289-9
Text printed in the United States on recycled paper at Courier in Stoughton, Massachusetts.
First printing, October 2009
To my loving and lifetime partner, Leigh, who has seen me through yet another book and without whom this book would not have been written.
—Alan Shalloway
To my darling family—Genny, Ruth Ann, Meredith, and Ben—for so graciously tolerating the travel required to amass the stories that helped create this book.
—Guy Beaver
To Dr. Bob and Sharon Foote, Lorien, Dedra, and Heather. Mentors in life, in faith, and in thought. Sola gloria Dei.
—Jim Trott
Contents
The Software Development Pendulum
Critique the Process, Work Together
Lean Provides More than Beliefs
PART I Extending Our View beyond Projects
The Software Development Team and Flow
Chapter 1 An Agile Developer’s Guide to Lean Software Development
Lean Applies to Many Levels of the Organization
A Quick Review of Some Lean Principles
Look to Your Systems for the Source of Your Errors
Minimizing Complexity and Rework
Eliminating Waste and Deferring Commitment
Using Iterative Development to Minimize Complexity and Rework
Reflections on Just-In-Time (JIT)
Using Value Stream Mapping to Get to True Root Cause
Chapter 2 The Business Case for Agility
Add Value to the Business Quickly
Promote Knowledge-Based Product Development and Better Project Management
Focus on Product-Centered Development
How to Create Real Value for an Organization
Manage the Organization’s Resources
Chapter 4 Lean Portfolio Management
The Challenge of Selecting Projects
Project Portfolios Are Idea Inventories
Should We Avoid Delays by Batching Project Analysis?
Should We Avoid Delays by Increasing Releases?
With Existing Systems, Plan to Use Incremental Delivery
The Benefits of Lean Portfolio Management
Line of Sight to Business Needs
Estimating and Tracking Progress
PART II Lean Project Management
Defining a Method While Not Being Restricted by It
Principles and Practices Open the Door for Professionalism
Misunderstandings, Inaccurate Beliefs, and Limitations of Scrum
Misunderstandings Commonly Held by New Scrum Practitioners
Scrum Beliefs We Think Are Incorrect
Limitations of Scrum That Must Be Transcended
Unfortunate Consequences of These Beliefs
Lean Thinking Provides the Necessary Foundation
Introducing Scrum#—Scrum Embedded in Lean Thinking
Introducing Kanban Software Engineering
Managing the Work in the Kanban Team
Chapter 6 Iteration 0: Preparing for the First Iteration
Chapter 7 Lean-Agile Release Planning
Transparent and Continuous Planning
Example: Release Planning Session
2. Prioritize Features, Left to Right
3. Split Features Using the MMF Perspective
4. Estimate the Value of Features
5. Estimate the Cost of Features
Chapter 8 Visual Controls and Information Radiators for Enterprise Teams
Visual Controls and Information Radiators
Product Vision: Providing the Big Picture
Product Backlog with Release Plan
The Visual Control for Multiple Teams
Establishing Clear Line of Sight
Managing Dependencies with Visual Controls
How to Tell If You Have a Good Visual Control
Chapter 9 The Role of Quality Assurance in Lean-Agile Software Development
QA at the End of the Cycle Is Inherently Wasteful
Improve Results by Moving QA Up Front
When the Product Champion Will Not Answer Your Questions
Executable Specifications and Magic Documentation
Acceptance Test-Driven Development
Chapter 10 Becoming an Agile Enterprise
The Importance of Continuous Process Improvement
Chapter 11 Management’s Role in Lean-Agile Development
Lean-Agile’s Balanced Approach to Management
Create Knowledge within the Team
Agile Software Development Is Not Anarchy
Lack of Management May Equal Lack of Success
Improving Management with Lean Thinking
Chapter 12 The Product Coordination Team
Getting Teams to Work Together
The Challenge of Coordinating Teams
Product Coordination Team Membership
Product Coordination Team Guidelines
Chapter 13 Software Architecture and Design’s Role in Lean-Agile Software Development
Avoiding Over- and Under-Design
The Role of Design in Software
The Role of Management in Software Design
PART III Looking Back, Looking Forward
Toyota: The First Great Example of Lean
Insights from Lean-Agile Coaches
Focusing on One Project at a Time
Initiating Fewer Projects Instead of Imploring Teams to Work Better
Knowing Where You Are: Minimum Releasable Features
Priorities and Work-in-Process
The Mantra of Lean: Fast-Flexible-Flow
An Example of Fast-Flexible-Flow
Appendix A: Team Estimation Game
Appendix B: A Model of Lean-Agile Software Development
Series Foreword. The Net Objectives Product Development Series
Alan Shalloway, CEO, Net Objectives
If you are like me, you will just skim this foreword for the series and move on, figuring there is nothing of substance here. That would be a mistake. Unless you have read this foreword in another book in the series, please take a moment with me at the outset of this book (if you’ve already read a foreword from another book, please skip a couple of pages to This Book’s Role in the Series).
I want you to consider with me a tale that most people know but don’t often think about. That tale illustrates what is ailing this industry. And it sets the context for why we wrote the Net Objectives Product Development Series and this particular book.
I have been doing software development since 1970. To me, it is just as fresh today as it was four decades ago. It is a never-ending source of fascination to me to contemplate how to do something better, and it is a never-ending source of humility to confront how limited my abilities truly are. I love it.
Throughout my career, I have also been interested in other industries, especially engineering and construction. Now, engineering and construction have suffered some spectacular failures: the Leaning Tower of Pisa, the Tacoma Narrows Bridge, the Hubble telescope. In its infancy, engineers knew little about the forces at work around them. Mostly, engineers tried to improve practices and to learn what they could from failures. It took a long time—centuries—before they acquired a solid understanding about how to do things.
No one would build a bridge today without taking into account long-established bridge-building practices (factoring in stress, compression, and the like) but software developers get away with writing code based on “what they like” every day, with little or no complaint from their peers. Why do we work this way?
But this is only part of the story. Ironically, much of the rest is related to why we call this the “Net Objectives Product Development Series.” The Net Objectives part is pretty obvious. All of the books in this series were written either by Net Objectives staff or by those whose views are consistent with ours. Why Product Development? Because when building software, it is always important to remember that software development is really product development.
By itself, software has little inherent value. Its value comes when it enables delivery of products and services. Therefore, it is more useful to think of software development as part of product development—the set of activities we use to discover and create products that meet the needs of customers while advancing the strategic goals of the company.
Mary and Tom Poppendieck, in their excellent book, Implementing Lean Software Development: From Concept to Cash (2006), note:
It is the product, the activity, the process in which software is embedded that is the real product under development. The software development is just a subset of the overall product development process. So in a very real sense, we can call software development a subset of product development. And thus, if we want to understand lean software development, we would do well to discover what constitutes excellent product development.
In other words, software in itself isn’t important. It is the value that it contributes—to the business, to the consumer, to the user—that is important. When developing software, we must always remember to look to what value is being added by our work. At some level, we all know this. But so often organizational “silos” work against us, keeping us from working together, from focusing on efforts that create value.
The best—and perhaps only—way to achieve effective product development across an organization is a well-thought-out combination of Lean principles to guide the enterprise, Agile practices to manage teams, and technical skills (test-driven development, design patterns). That is the motivation for the Net Objectives Product Development Series.
Too long, this industry has suffered from a seemingly endless swing of the pendulum from no process to too much process and then back to no process: from heavyweight methods focused on enterprise control to disciplined teams focused on the project at hand. The time has come for management and individuals to work together to maximize the production of business value across the enterprise. We believe Lean principles can guide us in this.
Lean principles tell us to look at the systems in which we work and then relentlessly improve them in order to increase our speed and quality (which will drive down our cost). This requires
1. Business to select the areas of software development that will return the greatest value
2. Teams to own their systems and continuously improve them
3. Management to train and support their teams to do this
4. An appreciation for what constitutes quality work
It may seem that we are very far from achieving this in the software-development industry, but the potential is definitely there. Lean principles help with the first three and the understanding of technical programming and design has matured far enough to help us with the fourth.
As we improve our existing analysis and coding approaches with the discipline, mindset, skills, and focus on value that Lean, Agile, patterns, and test-driven development teach us, we will help elevate software development from being merely a craft into a true profession. We have the knowledge required to do this; what we need is a new attitude.
The Net Objectives Product Development Series aims to develop this attitude. Our goal is to help unite management and individuals in work efforts that “optimize the whole”:
• The whole organization Integrating enterprise, team, and individuals to work best together.
• The whole product Not just its development, but also its maintenance and integration.
• The whole of time Not just now, but in the future. We want sustainable ROI from our effort.
This Book’s Role in the Series
While Scott Bain’s Emergent Design: The Evolutionary Nature of the Software Profession dealt with how to raise the bar in technical practices, this book is about how to raise the bar in product and project management. Both books, as I suspect all books in the series will be, are based on the belief that there are laws (rules) that must be followed to be effective and efficient.
As Agile has matured, we’re finding it useful to go beyond the mere mandate of building in stages and having teams solve their own problems. While both are sage advice, more is needed as our products become more complex. Management needs to become more intimately involved in solving the problems teams face. And although the development teams are the ones who actually deliver the value, they are not empowered to solve the organizational and cultural problems that get in their way.
We believe that Lean thinking provides a new way for management and teams to work together. We further believe that the next generation of Agile methods will be those that promote this cooperative effort instead of being neutral at best and negative at worst. This book is therefore about raising software development closer to a professional level throughout the organization.
The End of an Era, the Beginning of a New Era
I believe the software industry is at a crisis point. The industry is continually expanding and becoming a more important part of our everyday lives. But software development groups are facing dire problems. Decaying code is becoming more problematic. An overloaded workforce seems to have no end in sight. Although Agile methods have brought great improvements to many teams, more is needed. By creating a true software profession, combined with the guidance of Lean principles and incorporating Agile practices, we believe we can help uncover the answers.
I hope you find this book series to be a worthy guide.
Preface
This book was born from need and from knowledge. The need is to expand the knowledge base of software development in both the management and process worlds so as to create a new base. Integrating Agile has transformed the software-development process in less than a decade. Although its mandate applies to all of software development, its focus typically has been on the teams directly involved in the development of software and on the projects they work on. As Agile has begun to transcend the early-adopter phase and move on to the early-majority phase, there are new challenges to address as Agile is applied to quite different situations.
• Larger organizations are attempting to adopt Agile for the first time.
• Organizations that are already using Agile are expanding the scale of their adoption.
• Organizations that are somewhat dysfunctional are starting to adopt Agile.
Extending Agile to these new situations creates the need for a better understanding of what Agile is and a broader set of tools to apply Agile. These two issues are surprisingly tightly related. Many Agile early adopters have learned from any number of excellent books that present a set of practices, mostly oriented around the team. Unfortunately, few of these books explain why Agile works. Rather, they are filled with excellent practices that embody Agile’s fundamental belief systems while providing a set of practices that work at the team level in many situations.
The wider adoption of Agility demands more. There is now a need for a greater scope of knowledge as well as an explanation of why the practices work. While almost all Agile methods sprang up independently of Lean thinking, Lean thinking provides insight into why Agile works. This is why most of Agile’s methods are compatible with Lean. True knowledge is realized when one can apply principles and practices together to form solid understanding. We use the term “Lean-Agile” for the approach described in this book because it represents our contention that for Agile to work most effectively, it must be applied within the context of Lean.
This book fills the need both to understand why Agility works as well as to expand its base of principles and practices in order to apply it to the enterprise. It builds on the work of others, most particularly, those of David Anderson, Kent Beck, Jane Cleland-Huang, Alistair Cockburn, Jim Coplien, Ward Cunningham, W. Edwards Deming, Mark Denne, Ron Jeffries, Daniel Jones, Michael Kennedy, Corey Ladas, David Mann, Bob Martin, Rick Mugridge, Taichi Ohno, Mary Poppendieck, Tom Poppendieck, Don Reinertsen, Peter Scholtes, Ken Schwaber, Jeff Sutherland, James Womack, Alan Ward, and so many others. This blend of Lean, Agile, XP, Scrum, and other disciplines creates the synergistic blend essential to providing answers, both deep and broad, that the enterprise requires.
I want to give particular thanks to a few people who have helped us personally in our endeavors.
• Mary and Tom Poppendieck for helping me get my start in Lean training. Both have been invaluable to my personal development with their combination of suggestions for improvement tailored by respect and compassion.
• Don Reinertsen for his kindness and encouragement, not to mention the amazing amount of knowledge that his books have conveyed to the community.
• David Anderson for his outspokenness and out-of-the box thinking. He’s been an inspiration to go further in my thinking than I have typically dared.
• Ward Cunningham. I know few people smarter than Ward, balanced with such an unassuming nature. His wisdom and manner have been invaluable.
• Our own Alan Chedalawada, who may not have contributed to the writing in this book, but whose ideas formed the basis for much of what we are presenting here that is new. Many of these ideas he first manifested in the real world.
• Our own Amir Kolsky and Ken Pugh for insights into the role of acceptance test-driven development.
While it may seem odd for one author to acknowledge another, I must acknowledge Jim Trott—both a close associate and one of my dearest friends. Without his encouragement, hard work, and efforts on keeping me focused, this book may not have happened.
Alan Shalloway
CEO, Net Objectives
Achieving Enterprise and Team Agility
Acknowledgments
With every book we write, we become more deeply impressed with how important it is to acknowledge those who have helped with its development. Such an effort is indeed the work of a community. The list of people to whom we are indebted is long.
The following people have been especially significant to us:
• Chris Guzikowski, Raina Chrobak, and Chris Zahn from Addison-Wesley, who never grew tired of encouraging us, kept us moving along, and made us look good.
• Our fellow consultants at Net Objectives: Alan Chedalawada, Scott Bain, Amir Kolsky, Ken Pugh, and Brenden McGlinchey, who sparred with us to develop these ideas.
• Doug May, who reviewed early drafts of this book on his own time and gave us such helpful and timely critique.
• Andrea Bain, our talented graphics designer who took our hen scratches and made them understandable.
• Vicki Rowland, the best editor we’ve ever worked with and who made countless improvements to our grammar.
• Peter Alfvin, who often encouraged Alan to speak up about our unconventional ideas in public. This helped refine many of our ideas.
• Lorien Trott, who so ably and accurately prepared our manuscripts for the publisher with such a good attitude despite the many changes.
• And especially Leigh, Genny, and Jill, our patient wives who put up with us and encouraged us in our dream of this book. After being asked, “Are we done yet?” for the 3327th time, we are glad we can finally answer, “Yes!”
Finally, we have received many deep insights from our reviewers and students along the way. In particular, we recognize: Robert Bogetti, Domingo Chabalgoity, Michael Cottmeyer, Marc Evers, Paddy Healey, Alina Hsu, Kay Johnson, Mark Needham, Richard Karpinski, Armond Mehrabian, Paul Oldfield, Rob Park, and Tathagat Varma. We could not have done this without you!
About the Authors
Alan Shalloway is the founder and CEO of Net Objectives. With 40 years of experience, Alan is an industry thought leader. He helps companies transition to Lean and Agile methods enterprise-wide as well as teaches courses in Lean, Kanban, Scrum, Design Patterns, and Object-Orientation. Alan has developed training and coaching methods for Lean-Agile that have helped his clients achieve long-term, sustainable productivity gains. He is a popular speaker at prestigious conferences worldwide. He is the primary author of Design Patterns Explained: A New Perspective on Object-Oriented Design and Lean-Agile Pocket Guide for Scrum Teams and is currently writing Essential Skills for the Agile Developer. He has a Master of Science in computer science from M.I.T. as well as a Master of Science in mathematics from Emory University.
Jim Trott is a senior consultant for Net Objectives. He has used object-oriented and pattern-based analysis techniques throughout his 20-year career in knowledge management and knowledge engineering. He is the co-author of Design Patterns Explained: A New Perspective on Object-Oriented Design and the Lean-Agile Pocket Guide for Scrum Teams. He is a trainer and coach in the areas of reflective practices, knowledge management, and process improvement and is a knowledge-management consultant for international relief and development agencies. He has a Master of Science in applied mathematics and a Master of Business Administration from the University of Oklahoma and a Master of Arts in intercultural studies from Hope International University. An Associate Technical Fellow of a large aerospace company, he has also worked in the energy industry, banking and finance, software development, and artificial intelligence.
Guy Beaver is Vice-President, Enterprise Engagements and a senior consultant for Net Objectives. He is a technology executive with a track record of success in Lean-Agile implementations in large, mid-sized, and start-up organizations. He is a recognized expert in Lean, Agile, and Scrum technical development with a proven ability to lead, manage, and motivate organizations to realize significant improvements in productivity and quality. He has more than 25 years of experience in software engineering and IT across several industries, including financial services, defense, and healthcare. Guy has a Master of Science in physics from Wake Forest University.
Introduction
“We can’t solve problems by using the same kind of thinking we used when we created them.” —Albert Einstein
One of the goals of this book is to give you a better perspective on Lean and Agile and how to use them in software development. This requires an understanding of the roots of Agility, the software development “pendulum,” and the importance of paradigms and practices and of being pragmatic. Lean offers a way forward.
This book takes the reader beyond Agile’s standard practices by teaching how to incorporate Lean thinking into software development. Although Agile, as it is usually practiced, is effective at the team level, it gives little guidance on how it fits at the enterprise level. This is somewhat for historical reasons, as you will see. Lean-Agile is an approach to Agile software development using Lean principles and practices for guidance.
You can think of Agile in one of two ways: as a set of values and beliefs that leave it to the practitioners to decide how to apply them or as a set of practices that are suggested to manifest good results. Practitioners typically use a combination of both, believing the mandate of the Agile Manifesto and then using either Scrum1 or eXtreme Programming2 (or some of each) as the basis for their methods. The challenge with this approach is two-fold—one resulting from the roots of Agility and the other from the lack of a theoretical foundation for the Agile practices themselves—as we will discuss later.
1. Scrum is a popular Agile process created by Jeff Sutherland and Ken Schwaber. It is commonly used at the team level and is characterized by self-organizing, cross-functional teams doing iterative development in what are called sprints.
2. eXtreme Programming is an iterative development process for teams centered around several engineering practices. The most common of these is test-driven development, paired programming, and continuous integration.
How This Book Will Help You
This book aims to change how you look at software development. Doing so will enable you to solve seemingly intransigent problems with much less effort than you might have thought possible. One of our guiding principles is that we need to drive from business value: Deliver the value (software) that will provide the greatest return to the business by providing the greatest value to the business’s customers. For an IT development group, this could mean either internal or external customers.
Together, we will explore what software development actually is and how it must be managed. We will investigate ways to help our customers through the process of selecting what work to accomplish through development, deployment, and, ultimately, ongoing support and enhancement.
We will drive from principles throughout the book and provide a good many that you can apply in Lean Software Development. This book will not give you all the answers; instead, it will help direct your thinking so you can create answers that will work for you in your company, in your situation, for your customers, and with your products.
The Roots of Agility
The development of the Agile Manifesto (Beck et al. 2001) was a breakthrough event for the software industry. The manifesto, shown in Figure I.1, and its Twelve Principles, shown in Figure I.2, describe the essential ideology that underpins Agile software development.
Figure I.1 Manifesto for Agile Software Development
4. Copyright © 2001 Kent Beck, Mike Beedle, Arie van Bennekum, Alistair Cockburn, Ward Cunningham, Martin Fowler, James Grenning, Jim Highsmith, Andrew Hunt, Ron Jeffries, Jon Kern, Brian Marick, Robert C. Martin, Steve Mellor, Ken Schwaber, Jeff Sutherland, and Dave Thomas; this declaration may be freely copied in any form, but only in its entirety through this notice.
Figure I.2 Twelve Principles behind the Agile Manifesto
The Software Development Pendulum
The Manifesto is a strong statement. It is consistent with the intentions of most people in the software development industry. But it says that we must develop in a way that is different from the ways we have often tried in the past. It stands in opposition to the myth that the way to create quality, sustainable software is to conceive large plans and then use command-and-control management3 to realize them. When it was written, the Agile Manifesto presented a great opportunity for exploring new, better ways of developing software. Unfortunately, it also left a huge hole. It did not attempt to describe how to achieve the promise.
3. Apologies to military experts who properly use this term to mean vision at the top with implementation at the bottom. We’re using this term in the way most people interpret it—top-level people telling lower-level people how to get their job done.
This lack of instructions is not a shortcoming of the Agile Manifesto. The Manifesto’s purpose was to create a vision for a better way to develop software. It is instructive to look at the Manifesto in its historical context. During the decades preceding the Manifesto, the principles of and approaches to software management swung like a pendulum, between free-form and command-and-control, from little process to too much process. Each was responding to the challenges of the other.
In the 1960s, several large system failures demonstrated the need for both better engineering methods and better processes. Certainly, software development during this time was not an ad-hoc affair, but the industry was new and there was little experience with large-scale systems. In the 1970s, the idea of software as “engineering” surfaced. We began to use structured analysis and design, top-down programming, and structured programming (goto statements were considered bad form). Notably, the Waterfall model emerged. The industry was growing up, and standard practices for design, programming, and management arose. By the 1980s, PCs and fourth-generation languages enabled small projects to flourish. Small teams produced much more software than large teams did. Prototyping was popular. Speed was king. If you were the first, you were the best.
But quality often suffered. Speed to entry was so important that a product’s sustainability was often ignored. This led to different kinds of failures. Since it was easy for anyone to enter the market, the competitive edge of getting in first was lost if the product lacked quality. Failures in this era triggered an upsurge in rigorous process. The sense was that if we can’t do it ad hoc, then we’d better control it.
Tick tock. Tick tock. The pendulum continued to swing. Maybe even faster.
The 1990s brought us the Capability Maturity Model (CMM). Y2K dominated the last few years of the decade, emphasizing the need for planning ahead. But the ’90s also brought us the Internet, which again enabled small teams to have great impact. The dot-com boom brought rapid software development. Again, a proliferation of small teams found initial success but subsequently had difficulty maintaining the software that they had developed.
Now, the twenty-first century has given rise to Agility—small teams working with customers to develop software quickly. There have been many successes and there have been many failures.
And the pendulum continues to swing. What can we do to stop it? Or can we at least find a balance?
The Agile Manifesto was an attempt to find such balance. Let’s respect our teams. Let’s respect our customers. Let’s work with the business. Process can be good, but process that doesn’t help a team get its job done is not.
Unfortunately, the world is messy and the promise of the Manifesto has not been entirely realized. The Manifesto itself showed the potential, but it did not provide a means to stop the pendulum. In fact, it has been used to justify letting teams rule. We have mostly lost the enterprise view, because that view seems to lead right back down the path of command-and-control management. If the choice is between that and using teams with Agility, then abandoning command-and-control seems reasonable.
It is not either-or. There is a way to balance command-and-control with the need for effective teams. Lean provides the way. To see why, we must first examine the beliefs, principles, and paradigms on which we build our thinking.
Principles and Paradigms
A principle is a comprehensive and fundamental law, doctrine, or assumption. Principles may exist at the level of the individual, may be held by a community, or may even apply universally. For example, individual principles may relate to one’s integrity or one’s way of living. A communal set of principles might include moral or religious beliefs or a set of beliefs that the community accepts as the true way to be living. Universal principles are those that apply everywhere—beyond the effect of the beliefs of any set of individuals. Perhaps we should actually call these laws of the universe. Principles are often stated in the form of guidance since there is often a corresponding principle (law) that should be followed. For example, one of the best known Lean principles is “eliminate waste.” That’s not a law as much as it is something you should do, as a rule.
A paradigm is a combination of assumptions, values, beliefs, and practices that define how to view reality, how to look at a situation. It is a worldview that characterizes what is true. Paradigms tend to last a long time (consider how long people believed the earth was the center of the universe). Paradigms are shared by a particular community or group of people. In the software world, Waterfall, Scrum, and Lean-Agile each have their own paradigms, or way of looking at how to best build software.
Since a paradigm defines what is real and true for someone, changing one’s paradigm is quite difficult. It requires the individuals and their community to grapple with the underlying assumptions, values, and beliefs and assess whether the paradigm actually squares with what is indeed “real” or whether some shift is required.
The paradigms we hold constrain what we consider possible and shape what we do. Unexamined paradigms can therefore be very limiting.
A Pragmatic Approach
Software development professionals are pragmatists (pragmatism is part of our worldview). We favor what works over what is represented as theoretically “correct.” It is not that theory is bad, but theory must be grounded in real work if we are going to embrace it.
With that in mind, we would like to suggest taking a pragmatic approach to evaluating the essential paradigms we, as software developers, hold. That approach is to use the scientific method in whatever we do: Propose a hypothesis and then run an experiment to validate or invalidate it. If the experiment supports the hypothesis, then we have some evidence that the hypothesis is correct. If it doesn’t, then the hypothesis is incorrect and must be modified.
We suggest that in the software development world, our processes must be consistent with our hypotheses about the best way to practice software development. If we get good results, we have evidence that our process (that is, our hypothesis) is good. If we get poor results, our process needs updating.
Critique the Process, Work Together
Let’s be clear: This is all about critiquing the process, not the people involved. How many teams have run into problems because they are following a poor process and yet management, being overly committed to the process, blames the people? Assuming the process is right, they believe “if only the people had done it right, it wouldn’t have been such a disaster.”
Or how many projects have failed because teams decide to follow their own approaches regardless of the larger needs of the business? They assume management is just getting in the way—bureaucrats who must be worked around.
It seems that the tendency is for management to over-focus on process while teams underestimate its value. One side sees management as crucial to making the process work; the other wants to be protected from management’s command-and-control mentality so that they can just get their work done. And so they go back and forth, not working in concert.
What we need is a new attitude about process and how to manage process. Processes must be designed to assist the team in achieving management’s goals. Processes help the team get its job done: They represent accountability among team members about how they will work. The team is the steward of its processes—creating, sustaining, and improving them so that the team can improve constantly. Processes are dynamic: They are the team’s baseline for change.
Lean Provides the Way Forward
Is this possible? Yes! Lean provides the principles we need to do this. And we will not follow these principles blindly. Blind faith doesn’t work. Instead, we will use Lean as a guide and use our own experience to refine our own process.
If you have been building software for a few years, we invite you to use the hypothesis-and-test approach yourself: Run “backward-looking experiments,”5 that look back over your own past experiences to validate or invalidate the process we are developing. This is a lot more pragmatic and a lot less painful than trying new processes on future projects. You will be able to verify relatively quickly whether the process works.
5. A backward-looking experiment is a term Alan Shalloway coined to mean looking into your past to validate or invalidate an hypothesis made in the present. For example, if someone says “coding conventions help” he is actually postulating that coding conventions result in better code. You can actually look into your past to see when that was true (adding evidence to the hypothesis) or when it was false (disproving the hypothesis). If you disprove the hypothesis, you can modify it with a condition to see if there is a set of circumstances that would make it true. This enables us to learn about and test our understanding by taking advantage of our past experience.
As we do this, we will be building a pragmatic “theory” about why and how software development works. We recognize the truth in Jan L.A. van de Snepscheut’s or Yogi Berra’s comment “in theory, theory and practice are the same, but in practice, they are different.” We also believe Kurt Lewin’s notion that sometimes “there is nothing more practical than a good theory.” In other words, do not follow theory when it does not match practice. But when you are not sure what to do, an understanding of why your practices work may give you guidance in unfamiliar situations.
This pragmatic approach embraces the principles (or laws) that we have discovered work in all situations. For example, the principle that overloading an individual with work, that is, giving her many tasks to do at the same time, degrades her performance.
Principles lead to many practices. However, practices must change depending upon the context, or situation, in which they are used. Relying totally on principles may not work unless the principles are proven. Relying totally on practices will work only if you are in situations you’ve been in before. Effectiveness requires a proper blend of proven principles with practices appropriate for the situation in which they are being used.
Evaluating Paradigms
As we begin this approach, let’s look at some of the core beliefs upon which the Waterfall model and the Agile framework are based. These are described in Figures I.3 and I.4. Are these universal principles? Or are they unexamined paradigms—rules that just must be followed?
Figure I.3 Core beliefs of Waterfall
Figure I.4 Core beliefs of Agile
We believe that the core beliefs of Agile are more helpful than the core beliefs of Waterfall. Agile’s beliefs are helpful, but they are not enough. To follow them effectively, more is required. That is where Lean comes in.
We Do Not Know It All
Although software development is not exactly like other types of product development, we can still learn a lot from how other industries approach product development. In particular, Lean gives us a lot of information, based on decades of experience, that can be particularly useful to Agile teams. In fact, Scrum, one of the more popular Agile methods, is based on Lean principles. Unfortunately, an understanding of Lean is not widespread in the software community. It is unfortunate because teams lose out on the potential guidance that Lean offers. Moreover, without a grounding in Lean, software developers often lack the basis for explaining to management why certain practices would be useful. Lean provides a new set of beliefs, shown in Figure I.5. The question still remains: Even if these beliefs are true, how do we manifest good practices that are consistent with them?
Figure I.5 Core beliefs of Lean
Of course, merely believing something doesn’t make it so. It is worth looking at the beliefs presented here and then deciding which ones represent actual principles. We suggest using backward-looking experiments for this.
Lean Provides More than Beliefs
Fortunately, Lean provides more than a paradigm and a belief system. It provides a set of principles in its own right as well as many practices based on them. While these practices cannot usually be taken straight from Lean (since practices must change depending upon the context in which they are used), Lean principles and practices can be readily adapted to software development. By learning these principles and practices, one can manifest the intention of the Agile Manifesto—developing software effectively. And we can do it at both levels—enterprise and team.
We will see that Lean provides a paradigm of management in which managers are not encouraged to command and control teams and developers are not required to insist they are craftsmen who cannot and should not be managed. Rather, Lean provides a paradigm under which managers and developers can work together toward a common goal—providing the best return on software development efforts. Lean provides such a paradigm through its focus on the process by which the team works—but a process that must be the best one for the team to get its job done. Process is no longer something imposed on the team, but rather something owned by the team to make its work more productive as well as more enjoyable.
Lean combines this management paradigm with concepts, tools, and practices that give both sides a way to work together and improve visibility to management, direction from management, and team productivity.
Going beyond Lean
Of course, Lean is not all there is from which to pull. But our experience is that it is consistent with other useful paradigms, beliefs, and principles that come from other disciplines. For example, we learn from the building-architecture discipline and the software-design-patterns community that we should develop products by starting with the big picture. That is, don’t try to create a product by building it from small pieces. Keep the big picture in mind. This, unfortunately, is a lesson many Agile practitioners and consultants have long ignored (probably due to Agile’s heritage, which sprang up on smaller projects).
In this book, we incorporate what may be non-Lean practices but they are otherwise consistent with a central principle of Lean: “Optimize the whole.” In particular, we’ll see this in these areas:
• An enterprise focus instead of a team focus both for product-portfolio management and for team coordination, thereby providing a working alternative to Scrum-of-Scrums.
• A product focus instead of a project focus (where projects are enhancements to products).
• Managing requirement elicitation from the big picture instead of starting with stories and combining them into epics and themes.
• Driving release planning from business value instead of trying to manage the effective release of a collection of stories.
Summary
This introduction explored the roots of Agility, starting with the Agile Manifesto, its principles, and its historical context in the swing between management command-and-control and development teams wanting to apply their local knowledge to get work done. What is needed is a proper understanding of process as both/and: both as a tool for management and a responsibility of the team to steward what it knows.
Getting to this better understanding involves examining core paradigms, principles, and practices that everyone in software development holds. Lean-Agile offers a thinking practice to help form a better way of understanding. It is based on the solid foundation of Lean thinking and is entirely consistent with Agile practices.
Try This
These exercises are best done as a conversation with someone in your organization. After each exercise, ask each other if there are any actions either of you can take to improve your situation.
• Look at the beliefs of Waterfall listed in Figure I.3. Which of these are true?
• Look at the beliefs of Agile listed in Figure I.4. Which of these are true?
• Look at the beliefs of Lean thinking listed in Figure I.5. Which of these are true?