Within every business, there is a desire for rapid change to meet customer demands. Such changes usually involve changing supporting IT systems. When large business changes are required, the accompanying IT changes tend to be significant, too. However, all too often, these big projects hit problems, run over budget, are delayed, or simply get cancelled. Even in 2006, 65% of IT projects failed on one of these counts.[1] Large projects have an even poorer success rate. Such odds are very worrying when the stakes are very high. This book identifies the fundamental issues at the heart of the IT industry’s current approaches and provides a new way forward. All people involved in large-scale business and IT change should read this book.
The IT industry has many key dates, but the introduction in 1964 of IBM’s new-generation mainframe, called the System/360, marked the start of a new era. Until that point, buying a new business computer meant rewriting your existing software. The System/360 changed all that with the introduction of a family of compatible computers and associated devices: A program that ran on one would run on any. The industry followed suit with equivalent products, and the nature of IT changed in one fell swoop.
IT investments could now be easily preserved. The programs that ran on the System/360 still run on IBM’s mainframe platforms today.
This was an imperceptible change at first, but it was a hugely significant milestone. At this point, IT complexity started accumulating within the enterprise. Systems grew with the business. Thousands of person-years of time, effort, and money flowed into these IT systems. They got complex. They became elephants.
In the meantime, IT fashions came and went. Over the years, the original structured programs have been augmented by object-oriented programming, wrapped by component-based development, and advertised by Service Oriented Architecture (SOA). Each of these movements has had its own strategy for dealing with the complexity, but none ever really took it to heart.
Today’s IT systems are so complex that they simply defy everyday comprehension, spilling out of our minds as we try to get our heads around them. Responsibility for maintaining them is split among a variety of skilled groups and myriad products and programs that coexist to support the functions of the enterprise. To deal with this Hydra, we draw high-level architecture diagrams that comfort us by making things look simple. These diagrams are an illusion, a trick, a facade. They are, at best, approximations for easy consumption and high-level communication. At worst, they instill false optimism about our ability to make changes to that complexity.
Such “fluffy cloud” diagrams cannot hide genuine complexity forever. To achieve your business goals and change those systems, you must understand, communicate, and harness the real complexity. No one can understand the whole beast, so vast amounts of well-coordinated teamwork and unambiguous communication are required to complete such tasks. This combination of high levels of complexity and the need for clear communication of that complexity among hundreds of individuals destroys big projects.
IT systems are generally not implemented on Greenfields any more. The accumulated complexity since 1964 means that the environment for most big IT projects is one of immense challenge, entangled in an almost uncountable number of environmental constraints.
This is the underlying reason for the demise of most large-scale IT projects. Only 30% of large IT projects succeed.
Big projects are usually executed on “contaminated” sites, where you need to be careful of where and how you build; a change in one place can ripple through to other systems in unexpected ways. Such sites are more brown than green, and the IT industry needs to adopt a Brownfield-oriented approach to address them successfully.
This book introduces such a Brownfield approach and explains why current methods are still essentially Greenfield. It is specifically written for people who want to change their business and know that they can do it only by building on what has gone before. If any of the following is true, this book is for you:
You are a CIO, CTO, IT director, project executive, project director, chief architect, or lead analyst who is contemplating a significant change in your IT landscape.
You cannot afford to replace your whole IT landscape.
Your systems talk to a fair number of systems outside your direct control.
You would like to reengineer your existing IT environment so that it will remain flexible for the future.
You are deeply unhappy with the current failure rates of large IT projects.
You are contemplating sending a significant part of your IT development and testing work off-shore.
Eating the IT Elephant was written by two full-time Executive IT Architects from IBM who can and have ticked every single one of those boxes on a number of occasions. We have been accountable for the technical direction and day-to-day implementation of some of the largest systems integration and reengineering projects that IBM has undertaken. We believe strongly that existing Greenfield development approaches are an increasingly poor means of addressing today’s business problems through IT solutioning. To be blunt, we have a number of years of hard-won experience, and we have grown tired of the recurring problems of IT delivery. In recent years, we have deliberately sought a different approach; the following pages detail the fruits of our labors and that of our colleagues. Heretics we might be, but pragmatists we are also, and, hand on heart, we can say that the insight we share here has significantly accelerated and simplified a number of recent IBM engagements.
We don’t think the high failure rate of major IT projects is doing our industry any favors and would like to popularize the approach that has served us well. If we can help mitigate the impact of the unavoidably complex IT environment and knock down some big project communication barriers, we believe that success rate will improve.
This book is not a technical manual nor a cookbook; it does not contain a single line of code, and we have tried to minimize the use of technical diagrams and jargon. This is a book about changing the way we approach large and complex business and IT reengineering projects.
To make the book as accessible to as many people as possible, we have split it into two parts.
Part I is for all readers. Initially, it defines what is wrong with large-scale IT projects and determines the root cause of failure (see Chapters 1, “Eating Elephants Is Difficult,” and 2, “The Confusion of Tongues”). The heart of the book (Chapters 3, “Big-Mouthed Superhero Required,” and 4, “The Trunk Road to the Brain”) concentrate on defining an alternative solution—an Elephant Eater—and the Brownfield approach that goes with it. In Chapter 5, “The Mythical Metaman,” we look at the new species of businesses that emerge as a result.
Part II explains the technical and practical aspects of Brownfield for someone who might want to implement such an approach. It starts by analyzing existing Elephant Eating techniques (see Chapter 6, “Abstraction Works Only in a Perfect World”) and explains why Brownfield is different (see Chapter 7, “Evolution of the Elephant Eater”). In Chapters 8, “Brownfield Development,” and 9, “Inside the Elephant Eater,” we look inside the Elephant Eater and at some of the new technologies that have been used to implement it. The book concludes by explaining how the Brownfield approach can be implemented on a project and the benefits it can bring (see Chapter 10, “Elephant Eater at Work”).
For those who take the key messages on board, a wealth of technical information has already been published that will enable any organization to adopt the core technologies that we have used (or equivalent ones) to implement Brownfield in their own way (see the “Endnotes” sections of Chapters 8 and 9). We hope that enabling business and IT change via a new project approach, not technology, is at the heart of this book.
Chapter 1, “Eating Elephants Is Difficult,” introduces the metaphor that performing a large IT project can be compared to eating an elephant. It looks at why big projects fail and provides best practices on how to overcome some of the common reasons for failure.
Chapter 2, “The Confusion of Tongues,” explains why this accumulated IT complexity is the root cause of failure, focusing on the human communication problems it creates. It goes on to specifically examine the “great divide” between business and IT that compounds the problem.
Chapter 3, “Big-Mouthed Superhero Required,” introduces the core concepts of Brownfield. It looks at how Brownfield can be implemented to create an efficient Elephant Eater.
Chapter 4, “The Trunk Road to the Brain”: We despair at IT professionals’ inability to communicate as effectively and efficiently as those in other similar professions (such as real architects). Chapter 4 describes how the Brownfield approach combined with the VITA architecture opens up new forms of communication, remote collaboration, and visualization of complex IT problems.
Chapter 5, “The Mythical Metaman”: The first part of the book concludes with an examination of the likely impact of Brownfield. It forecasts a new breed of businesses that are infinitely more customer focused and agile than today’s and explains how such businesses might come into being.
Chapter 6, “Abstraction Works Only in a Perfect World”: This more technical half of the book opens by defining the characteristics of an Elephant Eater. It considers existing “Elephant Eating” approaches and notes that they tend to compound project difficulties via their extensive use of decomposition and abstraction.
Chapter 7, “Evolution of the Elephant Eater,” looks at Brownfield’s technical and project roots, and explains its key differences from previous ideas. It ends with some likely scenarios and real-life project examples for which Brownfield has been or could be especially beneficial.
Chapter 8, “Brownfield Development,” introduces how the Brownfield development approach can be deployed on a project. It shows how to strike a new balance between Agile- and Waterfall-based development techniques and provides some of the best elements of each. It also describes the core phases of Survey, Engineer, Accept, and Deploy, and states the benefits of the approach.
Chapter 9, “Inside the Elephant Eater”: If Chapter 8 described what happens on a Brownfield project, Chapter 9 explains how it happens. This chapter looks inside the workings of an Elephant Eater and explains how it eats the elephant. The chapter also serves as an easy-to-read introduction to the new semantic technologies that underpin Web 2.0 and the semantic web.
Chapter 10, “Elephant Eater at Work”: The book concludes with a look at the practical applications of the Elephant Eater and how it can help solve some of today’s most difficult IT problems. This chapter includes a summary of the key benefits of the Brownfield approach.
We hope that you will enjoy reading this book as much as we enjoyed writing it. If you’d like to see more, go to the website www.elephanteaters.org. Additionally, if you would like to see more of the dynamic nature of Brownfield, there are two exhibitions in Second Life. One Second Life site is dedicated to the book [Cypa 30,180,302]. The other Second Life site is dedicated to the use of Brownfield within IBM at [IBM 1 140, 150, 60]. We look forward to meeting you there.
| 1. | The initial CHAOS report from Standish Group in 1994 reported a 16% success rate for IT projects. This success rate has generally increased over the intervening years. In 2006, Standish Group reported 35% of IT projects being on time and within budget, and meeting user requirements. The only blip in that record appeared in 2004, when failure rates increased. Standish Group explained that in 2004 there were more big projects—they fail more often because they are often forced to abandon iterative development techniques. In 2007, a rival report to CHAOS by Sauer, Gemino, and Horner Reich looked at 412 projects. It found that more than 65% of IT projects succeeded, but it found no successful projects greater than 200 person-years. This book looks specifically at those large projects. Hayes, F. Chaos Is Back. www.computerworld.com/managementtopics/management/project/story/0,10801,97283,00.html. Krigsman, M. Rearranging the Deck Chairs: IT Project Failures. http://blogs.zdnet.com/projectfailures/?p=513. Rubinstein, D. Standish Group Report. www.sdtimes.com/article/story-20070301-01.html. Sauer, C., A. Gemino, and B. Horner Reigh. “The Impact and Size and Volatility on IT Project Performance.” Communications of the ACM 50 no. 11 (November 2007): 79–84. |