Agile’s Genesis

In the 1990s, software development was believed to be in crisis. They actually called it that: “The Software Crisis.” Software projects were over-budget, late, didn’t meet requirements, and—according to the oft-quoted and ominously named “CHAOS Report”—a third of them failed outright.

Agile wasn’t a response to this crisis. Far from it. Agile was a response to the response.

To bring software development under control, big organizations created highly detailed processes that defined exactly how software was to be created. Everything was tightly controlled so that no mistakes could be made. (In theory, anyway.)

First, business analysts would interview stakeholders and document the system requirements. Next, software architects would read the requirements documents and create detailed design documents specifying every component of the system and how they related to each other. Then programmers would convert the design documents to code. In some organizations, this was considered low-skill work—just a mechanical translation exercise.¹ Meanwhile, test leads would use the same documents to generate test plans, and when coding was done, armies of QA personnel would manually follow those test plans and report variances as defects. After each phase, everything would be carefully documented, reviewed, and signed off.

This approach was called “waterfall development” or “phase-gate development.” If it sounds like a ridiculous straw-man, well, consider yourself fortunate. Not every company used a heavyweight process in the ’90s, but it was widely recognized as a logical and sensible way to work. Of course you needed to define requirements, then design, then implement, then test. Of course you needed to document every phase. This was discipline. This was engineering. How else could you possibly succeed?

Born Out of Crisis

Big companies defined their processes in excruciating detail. Roles, responsibilities, document templates, change control boards… every aspect of development was defined and controlled. If a project didn’t succeed—and according to the CHAOS Report, two-thirds of them didn’t [xxx check]—it was because the process needed more detail, more documents, more sign-offs. It resulted in a massive amount of documentation. Martin Fowler called it “The Almighty Thud.”²

This wasn’t a great way to work. It was bureaucratic and dehumanizing. Skill didn’t seem to matter as much as adherence to process. Programmers felt they were interchangeable cogs in an impersonal machine. It didn’t even work all that well.

So several people independently developed simpler, slimmer, and less prescriptive methods for developing software. They were called ”lightweight methods” in contrast to the heavyweight methods used by big companies. These new methods had names like “Adaptive Software Development,” “Crystal,” “Feature-Driven Development,” “Dynamic Systems Development Method,” “Extreme Programming,” and “Scrum.”

By the late ’90s, these methods were attracting serious attention. Extreme Programming, in particular, saw an explosion of grass-roots interest among programmers. In 2001, 17 of the lightweight methodology proponents met at a ski resort in Utah to discuss unifying their efforts.

The Agile Manifesto

“I personally didn’t expect this particular group of [people] to ever agree on anything substantive,” Alistair Cockburn said later.³

And, in fact, after two days, they only accomplished two things: the name “Agile,” and a statement of four values (see Figure 1-1). Over the following months, over email, they hashed out twelve accompanying principles (see Figure 1-2).

This was the Agile Manifesto. It changed the world. So, as Alistair said, they did agree on something substantive after all.

Figure 1-1. Agile Values

Figure 1-2. Agile Principles

But there was no unified Agile method. There never has been, and never will be. Agile is three things: the name, the values, and the principles. That’s it. It’s not something you can do. It’s a philosophy. A way of thinking about software development. You can’t “use” Agile or “do” Agile… you can only be Agile. Or not. If your teams embody the Agile philosophy, then they’re Agile. If they don’t, they’re not.

The Essence of Agile

Martin Fowler has made a career out of turning complicated software topics into well-considered, even-handed explanations. His explanation of “The Essence of Agile Software Development”⁴ is one of the best:

Agile Development is adaptive rather than predictive; people-oriented rather than process-oriented.

Martin Fowler

Why Agile Won

In the first ten years after the Manifesto, Agile faced enormous criticism. It was “undisciplined,” critics said. “It could never work.” Another ten years after that, the critics were silent. Agile was everywhere, at least in name. Heavyweight waterfall methods were effectively dead. Some younger programmers had trouble believing anybody ever could have worked that way.

There’s actually nothing inherently wrong with the basic waterfall process. If you keep it slim and operate in a well-understood domain, waterfall can work well. The problem was the heavyweight processes big companies used. Ironically, the processes designed to prevent problems actually caused many of the problems organizations were seeing.

With software, it’s very difficult to imagine how it will work before you actually use it, and it’s even harder to think of absolutely everything your software needs to do. This is doubly true for people who aren’t actively involved with software development. As a result, it’s critically important to get working software in front of people as soon as possible. You need to get feedback about what’s missing or wrong, then change your plans based on what you learn. As the Manifesto says, “working software is the primary measure of progress.” Learning and responding to change are at the heart of what Agile is all about.

Those heavyweight processes put so much emphasis on process controls and documentation and sign-offs that they incurred a huge amount of delay and overhead. They took years to produce working software, and they had nothing concrete to show until near the end. Instead of welcoming change, they actively worked to prevent change. They actually had a dedicated part of the process, the Change Control Board, whose primary purpose was to say “no” to change requests. (Or, more accurately, “yes, but it will cost you.”)

All of this added up to projects that spent years in development before they had anything to show. When they did, it was too late and too expensive to make changes. They ultimately shipped software that didn’t do what customers needed.

Although there are a variety of approaches to Agile—and some of them are more about co-opting a popular name than following the actual philosophy—one thing they all have in common is a focus on making progress visible and allowing stakeholders make course corrections as they go. This seems like a small thing, but it’s incredibly powerful. It’s why we no longer hear about a Software Crisis. Software is still late. It’s still over budget. But Agile teams don’t spend years building failures. And that’s huge.

There’s more to Agile than just providing visibility. But this one thing? This was enough. It’s why everybody wanted Agile.

Cargo Cult Agile

At first, Agile was a grassroots movement. It was largely driven by programmers seeking better results and better quality of life. As it became more successful, Agile’s momentum shifted from the underlying ideas to hype. Rather than saying, “let’s get better results by adapting our plans and putting people first,” organization leaders started saying, “Everybody’s talking about Agile. Get me some Agile.”

The thing is, there is no “Agile” to go get. It’s just a set of values and principles. There are specific Agile approaches, such as Extreme Programming and Scrum, that will tell you how to be Agile, but you still have to be on board with the underlying philosophy.

And for a lot of organizations, that underlying philosophy—adapting plans and putting people first—is really, really foreign.

The tragedy of the cargo cult is its adherence to the superficial, outward signs of some idea combined with ignorance of how that idea actually works. In the story, the islanders replicated all the elements of cargo drops—the airstrip, the tower, the headphones—but didn’t understand the vast infrastructure that enabled airplanes to arrive.

The same tragedy occurs with Agile. People want Agile’s Cargo: better results, more visibility, fewer business failures. But they don’t understand the underlying philosophy, and often wouldn’t agree with it even if they did. They want to buy Agile, but you can’t buy an idea.

What they can buy is the outward signs of Agile. Stand-up meetings! User stories! Tools! Consultants! There’s lots of stuff labelled Agile, and plenty of people eager to sell it to you. It’s often labelled as “enterprise-grade,” which is a way of saying “don’t worry, you won’t have to change.” Uncomfortable ideas like “adaptive planning” and “people-centric” are ignored.

And that’s how you get a cargo cult. All the activity; none of the results. The Agile part is missing.

“At my old company they wasted a huge number of man hours in meetings.”

“[Agile] cost an entire team (30+) people their jobs as they produced almost nothing for almost a year.”

“All [Agile] means is developers get shafted when the project changes… the day before delivery.”

“Its because of Agile there are sh—ty developers and sh—ty products being shipped.”

“I can’t take this Agile crap any more. It’s lunacy.”

comments about Agile from around the Web

The name Agile is everywhere. The ideas of Agile aren’t. It’s become self-perpetuating; for many, the only Agile they know is Cargo Cult Agile.

It’s time to fix that. In the rest of this book, I’ll show you how to apply Agile ideas for real. Keep an eye out for the Cargo Cult Agilists shown in the margin. They’ll show you what not to do.

Ready? Let’s go.