Chapter 5. Going beyond Scrum

“It isn’t what we don’t know that gives us trouble; it’s what we know that ain’t so.”

—Will Rogers

Learning a New Way

There are many models that describe how people learn new skills. It is widely acknowledged that people go through different stages: beginner, basic competency, competent, advanced, expert, virtuoso, master. Transitioning from stage to stage involves starting with a basic set of principles and practices and then expanding on them as the learner becomes more adept. The model evolves by building precept upon precept.

For example, when learning how to drive a car in the United States, a new driver may be told a few principles, such as “first be concerned about safety” and “use your own judgment . . . don’t let others hurry you.” They will also be given a few practices, such as “drive on the right side of the road,” “don’t follow anyone too closely,” and “put on your blinker before you make a turn.” This is the basic set, the “100-level” for a beginner.

Before long, the beginning driver has to transcend these rules and understand the principles upon which they are based. “Driving on the right side of the road” actually is a manifestation of a principle: “Driving in the same direction as the other cars in your lane is safer than driving in the opposite direction.” But, even in the US, if you want to make a left turn on a two-lane, one-way road, you should do it from the left lane. So the rule, “drive on the right side of the road,” is insufficient.

The point is that to become adept at any new process, we need to keep adding principles and practices to what we already know. For some methods, this can be burdensome. If we have no guiding force, it can be overwhelming. Fortunately, Lean-Agile has a mindset that can give guidance without being burdensome. Lean-Agile presents a way of thinking to solve new problems as they come up as well as a way of organizing practices that have been established to work in particular situations (contexts).

Defining a Method while Not Being Restricted by It

In the introduction, we discussed how our industry seems to cycle between too much process and not enough process. Processes can be used to micromanage teams or can be so loose as to be useless to the organization. Teams react against or are cowed by rigid processes. But they also go astray when process is missing.

Product development requires discovery and feedback so that what is learned can be incorporated; it also requires systems so that what is learned can be incorporated efficiently. It requires both fire and a fireplace: bright people with passion and a structure to help focus them.

Lean-Agile balances the extremes of too much or not enough process by looking for a process that supports the team. Lean thinking assumes that most errors are systemic in nature. To resolve them, it is critical to understand the way we do our work. Team members must understand the process they are following. Lean thinking assumes that the purpose of the process is to support the team. The team understands their work and conditions better than anyone else; therefore, the team is responsible for their process. When the process dictates actions that don’t best fit the situation, the team must modify (improve) it.

Defining a Process

How do you define a process that supports teams? By balancing the following:

• Principles apply in all contexts; practices apply in only certain contexts.

• When learning something new, people are in transition and can learn at only a certain rate.

• Process definitions need to be updated as the team learns, and they should help people progress from beginner to expert.

We want a model that can be picked up fairly easily by beginning teams and that can then expand as the team learns and becomes capable of incorporating more knowledge into the model. This balance enables us to provide a definition of a methodology that is not too overburdening to new practitioners yet is rich enough for people once they gain experience.

This concept represents a significant break in thinking between Lean-Agile and many other Agile methods. eXtreme Programming (XP) started with a dozen practices and a few values. As it became more popular and people became more skilled with it, however, little was done to explain the principles on which XP was based.

Several practitioners went beyond the specified practices and used their intuition and experience to take XP beyond situations where the practices as specified would work. This was a good thing. Unfortunately, little effort was made to codify these new practices or the thinking behind them, which meant it was hard for others to transition quickly from beginner to expert. A lot needed to be relearned. This learning was expedited, of course, with coaching from others who had already undergone the transition. But coaches of this nature are often either unavailable (if one looks internally) or expensive (if one has to learn externally).

We see the same thing happening with Scrum. Scrum is propagating through the industry because it is easy for individuals or small groups of teams to adopt it. Scrum’s practices readily work inside organizations that have well-defined teams, good communication channels with customers or their representatives, few interruptions to support existing products (where the teams adopting Scrum are the only ones to support them), and there are not many projects in process at any one time (about one at a time per team). This is the context in which the basic Scrum practices work well.

However, most organizations comprising several teams do not work under these conditions. Very often, an organization decides to adopt Scrum and it creates special cases, such as the one just described, for one select team; the result is success with Scrum for that team. Unfortunately, when they try to expand Scrum to other situations in the organization, it does not work nearly as well because they haven’t first addressed the necessary core organizational issues.

Scrum, like XP before it, has responded to these challenges by relying almost entirely upon developers learning as they go. Developers are expected to “inspect and adapt.” This is good, but it’s not nearly enough. Both XP and Scrum have good belief systems and values but speak little of the principles underlying their practices.

Lean thinking uses a richer approach, the “Plan-Do-Check-Act” (PDCA) cycle. PDCA requires the team to do its work according to an explicit plan of execution—a “model” if you will, that is guided by experience, good principles, and lessons that others have learned. This plan becomes their standard process. They do their work according to the plan then stop and check their experience against the plan. Based on their observations, they decide how to adjust the plan—what to change and what to keep doing. Then they begin to plan again.

The difference between “inspect and adapt” or “sprint plan, execute, retrospect” and PDCA is that PDCA includes an explicit statement of the workflow that the team is using. We plan what and how we will do our work. We do that work. We check to see the results that this “model” achieved. We then act accordingly—planning again, as necessary. The former approaches leave the team to figure out for itself what to do based on their own intuitions and with little guidance. PDCA, built on a model of Lean principles, provides more specific guidance and enables us to check the validity of our understanding of the work involved.

PDCA also applies more widely; it is not limited to the development cycle but becomes part of the mindset in all aspects of the team’s work. A testing team can use PDCA in its work that is founded upon test-driven development (TDD) principles. A UI team adjusts its interview process after each set of user interviews based on what they got versus what they expected.

For software development to become a true profession, we must use work processes that are guided by good principles and by the experience of others while constantly being open to critique and learning.

Principles and Practices Open the Door for Professionalism

There is an analogy here with design patterns. In 1994, Gamma, Helms, Johnson, and Vlissides published their seminal book, Design Patterns: Elements of Reusable Object-Oriented Software. Most people have understood patterns as “solutions to recurring problems in a context.” That is good for the beginners’ level. But patterns are much more than that. Christopher Alexander, author of The Timeless Way of Building (1979) inspired the patterns community when he said that patterns are really about resolving the forces (or issues) that need to be resolved in recurring situations. Learning how to resolve these forces leads to a discovery of much deeper principles rather than mere solutions. This deeper understanding of patterns¹ can become a foundation for establishing a Lean-based thought process to use for designing practices that solve problems teams face.

1. See Design Patterns Explained: A New Perspective on Object-Oriented Design (Shalloway and Trott 2004).

The Lean-Agile approach to creating a model for undertaking software development is a combination of foundational principles, beginning practices, and a thought process that teams can use to expand on their knowledge and to incorporate lessons learned from others. This creates the basis for a level of professionalism in software process that heretofore has not been achieved.²

2. This is analogous to Scott Bain’s exhortations of creating a basis for the technical aspects of software development in Emergent Design: The Evolutionary Nature of the Software Profession (2008).

Knowing Where You Are

Any skill that is really useful in life takes time to master. Sometimes you make great progress when you are first learning a skill. Danger lies in thinking that your surface understanding is deeper than it is. Wise people keep pressing on to learn and improve so that they can handle the inevitable challenges. You need to be prepared when the crisis comes—that’s not the time to begin preparing.

As educators in several different areas (Lean, Agile, Kanban, Scrum, product management, design patterns, Test-Driven Development), we have seen the importance of clearing away misunderstandings before proceeding on to new concepts. While we never want to forget what we know, misimpressions about what we are learning can stand in our way.

We have chosen to discuss the misunderstandings of Scrum to illustrate the difference between Lean-Agile and other Agile methods because Scrum is widely used and reasonably well known. It represents much of the current attitude in the industry about Agile and is therefore representative of much of the industry’s thinking, particularly with new teams attempting to adopt Agile methods.

The following sections cover several beliefs we have encountered that impede learning Scrum effectively. The first involves misunderstandings about Scrum itself—things that people believe but that are not true about Scrum. The second takes on concepts that Scrum does seem to advocate but that we feel are not effective. Errors in understanding must be cast aside before the true intentions can be grasped. Whenever we discover limitations in our thinking, or in this case, the thought process we are learning (Scrum), that becomes a place where we can expand our thinking.

Scrum is a really useful approach. It seems simple and yet it, too, requires skill and determination. To be prepared, you have to understand its principles and practices so that you can adapt to and address the challenges you will face.

After we explore some misunderstandings we have witnessed in the industry and how to get beyond them, we will conclude with a few concepts that many Scrum trainers believe but that we do not agree with.³

3. We are not saying that experienced Scrum practitioners cannot succeed at the enterprise with Scrum. After all, Scrum is merely a framework for building software and the team members need to fill in the framework with their own knowledge. What we are saying is that many of the things Lean provides fit nicely into that framework and should be used. Also, experienced Scrum practitioners know when to break the rules, so to speak—that is, use their intuition. We believe many of the things they intuit on their own can be explained consciously with Lean thinking. Our experience is that intuiting solutions is good, but being able to explain why you did what you did is better.

Scrum Is a Framework

Scrum is a framework for creating an effective Agile development process. It is based on the belief that software development must be controlled by responding to feedback received during the course of development. That is, although software development is inherently empirical (you can’t predict it) you can control it with feedback. The more frequent the feedback, the more effective developers can be. Scrum suggests building software in stages—say, every two to four weeks. Assess where you are, reprioritize, and develop the next step. Doing this helps expose problems and impediments. And problems are not to be avoided, but rather to be solved. For example, a team may find it does not have enough contact with someone who can speak for their customers. This is a problem that needs to be solved (increase the level of contact) or the team’s efforts will be hampered.

Following Scrum means bringing problems to the surface, solving them, then moving forward until more problems surface, and then solving them. There is not a one-size-fits-all approach to this because each team—and the problem domain they work in—is different. They must learn to learn. They must also throw away any limiting beliefs.

Misunderstandings, Inaccurate Beliefs, and Limitations of Scrum

This section discusses a variety of beliefs about Scrum that we have heard from many Scrum practitioners, both new and experienced. These can be grouped into three categories as follows:

• Misunderstandings commonly held by new Scrum practitioners

• There is no planning before starting your first Sprint.

• There is no documentation in Scrum.

• There is no architecture in Scrum.

• Scrum beliefs we think are incorrect

• Scrum succeeds largely because the people doing the work define how to do the work.

• Teams need to be protected from management.

• The product owner is the “one wring-able neck” for what the product should be.

• When deciding what to build, start with stories: Release planning is a process of selecting stories to include in your release.

• Teams should be comprised of generalists.

• Inspect-and-adapt is sufficient.

• Limitations of Scrum that must be transcended

• Self-organizing teams, alone, will improve their processes beyond the team.

• Every sprint needs to deliver value to the customer.

• Never plan beyond the current sprint.

• You can use Scrum-of-Scrums⁴ to coordinate interrelated teams working on different products.

4. Scrum-of-Scrums is a method to coordinate activities among several teams that are working together. Each team sends a representative to a meeting that occurs as often as necessary—very often weekly or even twice a week.

• You can use Scrum without automated acceptance testing or up-front unit testing.

Misunderstandings Commonly Held by New Scrum Practitioners

These are only a few of the misunderstandings we have run across. We mention them because they are some of the most frequent and some of the most damaging.

There is no planning before starting your first Sprint

Many people think that Scrum says just to jump in on day one and build the first part of the system. Actually, Scrum acknowledges that some pre-planning is necessary. We will talk about this in great detail in chapter 6, Iteration 0: Preparing for the First Iteration.

There is no documentation in Scrum

Actually, Scrum doesn’t address this directly, but it suggests that there be no documentation unless there is business value for it. This does eliminate many types of documentation. As in all Agile methods, document things only when that documentation will actually be useful. Don’t write documentation simply because the process says to do so (and if your process does say this, you should change your process).

There is no architecture in Scrum

Again, Scrum doesn’t address this directly. Scrum is more about managing the team than defining the work the team should do. Chapter 13, The Role of Architecture in Lean-Agile Projects, discusses the proper use of architecture.

Scrum Beliefs We Think Are Incorrect

We have found that these beliefs either lower a team’s effectiveness or make it less likely that improvements will be made. This is not a complete listing, but these are what we see as the more common and harmful beliefs.

Scrum succeeds largely because the people doing the work define how to do the work

It is true that Scrum follows Lean’s mandate that the people doing the work define how the work is done. However, the biggest improvement that many teams achieve when they initially practice Scrum has little to do with this. Consider “teams” that

• Have to pull people from other parts of the organization to get all of their needed skills

• Work on many projects at a time

• Are not co-located

• Have to follow what amounts to bureaucratic policies that are counterproductive

Now, imagine that you are to pilot a Scrum project and are told:

• You will have a cross-functional team with all of the skills you need.

• You will work on only one project at a time.

• You team will be co-located.

• You will not have to follow bureaucratic polices that are counterproductive.

You probably would expect a great productivity increase. We have seen teams like this be three times more productive than other teams in the same company even when they work on the same types of projects and have the same level of personnel. We believe that Scrum’s first times-three improvement level is often because thrashing stops and delays are cut out. There may be cases when we’re unable to do iterative development to the extent we would like, but we still see a huge productivity increase by taking advantage of this.

We want to understand this because we can often make productivity improvements by implementing these items even if Scrum can’t be started. Misidentifying the cause of improvement can result in missed opportunities for further improvement.

Teams need to be protected from management

Many Scrum practitioners say that the team should be insulated from management. They misunderstand the Scrum mandate to protect the team from interruptions.

This misunderstanding is based on the experience in some organizations that management causes most of the interruptions and therefore the team must be protected from management. This creates an “us (developers) versus them (management)” conflict that has caused so many problems in the software industry. Some generally understood Scrum principles are closer to folklore than actually a part of Scrum. Many of these tend to undermine management.

The famous “chickens and pigs”⁵ story used by many Scrum practitioners encourages this attitude. The intent is to illustrate how some people on a project are committed to it while others are merely interested in it. Unfortunately, it is usually used as a way to keep management from being involved. Management is not an impediment to be removed—it’s an asset that keeps the entire enterprise moving in the right direction. Chapter 11, Management’s Role in Lean-Agile Development, describes attributes of good Lean-Agile management. Again, Lean provides a way for management and workers to work together. Lean’s directive that management supports the team while the team creates their process provides guidance here.

5. “Chickens and Pigs” is based on an old joke. A chicken and a pig are in a bar having a drink when the chicken says to the pig, “We should open up a restaurant.” The pig says, “Oh? And what will we serve?” The chicken responds with “ham and eggs.” The pig considers for a moment and then answers, “I don’t think so. While you would be interested, I would have to be committed!”

It is true that in some organizations teams won’t self-organize unless management steps back and allows them to. In this case, the Scrum Master must encourage the team to make decisions in the new vacuum that management’s disappearance has created. But this views managers as capable only of managing and not leading. Lean-Agile considers management’s role as one of leadership, which is a distinct difference from the way many Scrum practitioners view it.

Leaving this attitude unchecked can result in dysfunctional teams that never quite become effective due to thrashing and poor integration with other groups. At that point, only management’s involvement can bring the situation under control.

Beyond this, there are times when management is essential because teams are constrained by their own local concerns. For example, if reorganization is needed, management must be involved; if impediments are introduced by factors outside the team’s control, then management can help resolve them.

Management is a partner in improvement.

The product owner is the “one wring-able neck” for what the product should be

Actually, no one’s neck should be “wring-able”! The product owner is the keeper of priorities but the entire team is responsible for building a quality product. Lean provides a way for developers and managers to work together. It starts with the name of this role: To indicate that it is leadership—not ownership—that we need, Lean uses “product champion” instead of product owner.

The product champion leads the developer team in discovering what the customer truly needs, and she or he assists and guides the rest of the team in this discovery. The product champion and the team are responsible for the quality of the product. The product champion may be responsible for prioritizing stories, but the development team is no less responsible for the product as a whole.

Practical experience from the field suggests that the product champion role comprises a team of product managers, business stakeholders, business analysts, and client-facing personnel who are committed to providing the required service levels of feedback and validation so that the development organization can move quickly. Chapter 10, Becoming an Agile Enterprise, covers this in detail.

When deciding what to build, start with stories: Release planning is a process of selecting stories to include in your release

It is almost always better to start with the big picture. In particular, Agile analysis should be a progression from business capability to sets of features to stories to tasks. The concept of minimum marketable features, described in chapter 4, Lean Portfolio Management, is essential. If you are losing the big picture while working on little pieces, this misunderstanding may be why—you shouldn’t be starting with the little pieces. Lean’s principle to “optimize the whole” helps provide guidance here. Chapter 7, Lean-Agile Release Planning, describes this in greater detail.

Teams should be comprised of generalists

This is an overly simplistic view. If everyone on the Scrum team can do every task then it is definitely easy to manage how stories are built. In reality, many applications have complexities that require specialists, such as database analysts and developers of stress-test algorithms in aviation.⁶ What is really needed is a team that is organized so that it has all the skills it needs to complete the work in a short time. The more knowledge is shared, the better. But the guiding rule is that the team have the necessary blend of skills.

6. Some of these skills require a Ph.D. and years of experience—not an easy thing to replicate.

Inspect-and-adapt is sufficient

We discussed this earlier, but it bears repeating. Inspect-and-adapt is good and necessary but it is not sufficient unless it includes explicitly improving the process with which the team works. It is also necessary to incorporate learning and guidance from others and from past experience. The better model is Plan-Do-Check-Act. Having to relearn should be considered a type of lost knowledge.

Limitations of Scrum That Must Be Transcended

Scrum works extremely well for teams within functional organizations. Unfortunately, many people are trying to adopt Scrum in less than fully functional organizations. Although Scrum may help teams isolate themselves from dysfunction in the organization (which can lead to limited improvement), it is better for them to help the organization become more functional. While this is not necessary for an individual team to accomplish its work, it is necessary for multiple teams to work together effectively. Lean-Agile’s broader perspective can help here and it is essential if we are to achieve enterprise Agility.

Self-organizing teams, alone, will improve their processes beyond the team

This clearly relates to the prior misconceptions concerning management. Purely self-directed teams have a history of not succeeding well. Scrum teams should be self-organizing, not self-directing. Continuous process improvement can be accelerated by a partnership between teams and management. Although the Scrum Master can provide some of the leadership necessary to prod teams into self-assessment, it is not enough. In fact, the Lean Enterprise Institute (Womack and Shook 2006) described the crucial role that middle management plays, such as asking intelligent questions and establishing an environment in which there is both leadership and collaboration while avoiding being either an autocrat or a hands-off manager. Lean’s paradigm of management providing leadership to teams that continuously improve their process provides guidance here.

Every sprint needs to deliver value to the customer

Until the software is released, there is no value delivered, no matter what you do. While each iteration (which Scrum calls a “sprint”) should include something that can provide feedback, it is not necessarily true that the iteration is always for the customer’s benefit. There are cases when you need to learn something about the system. These situations don’t occur as often as some developers think—usually the biggest risk is in building what the customer doesn’t need. But there are times when not discovering something about the system now will cause you great problems later (such as redesign or higher integration costs). In these cases, building what can most mitigate your risk may be more appropriate. Lean’s principles of “optimize the whole” and “eliminate waste” provide guidance here. We don’t want to overbuild, thus adding waste, but at the same time, we must keep the big picture in mind.

That said, you risk losing interest and feedback from key business stakeholders if iterations fail to show verifiable progress against a road-map. Being able to deliver end-to-end slices of capabilities is a higher-level technical skill that cannot be achieved in a hand-off, legacy organization. It requires cross-functional teams working together. In making the tradeoff between delivery of infrastructure and verifiable business value, always lean toward the latter to ensure that business and stakeholders stay attentive and engaged. They should serve as a natural constraint to “building what is not needed” or looking too far ahead.

Never plan beyond the current sprint

Oh, if only this were possible! On small teams and small projects, it might be; however, as the effort gets larger, it becomes increasingly difficult. An iteration backlog (visual control) sophisticated enough to handle multi-iteration and multi-team stories can manage this, and it isn’t really that complicated. We talk about this in chapter 8, Visual Controls and Information Radiators for Enterprise Teams. Lean’s larger view together with its mandate on eliminating waste helps here again—look as far as you need, but no farther. Dependencies among teams that work at different rates require looking ahead to make sure their efforts are well coordinated.

You can use Scrum-of-Scrums to coordinate interrelated teams working on different products

Scrum-of-Scrums is a great practice for coordination. However, when the different teams involved in the Scrum-of-Scrums have different purposes, motivations, or driving metrics, Scrum-of-Scrums simply does not work well. When the pressure is on and when teams have different motivations, they tend to home in on solving their problems. It is human nature to focus on those closest to us—so when we try to coordinate teams, naturally we will do what is in our own team’s best interest.

Lean can provide a bigger view. We all know the challenge of creating teams from individuals—we must provide a common goal. Creating an organization from individual teams has the same problem. Having a product coordination team that reaches across teams can solve this. Chapter 12, The Product Coordination Team, provides a better alternative.

Another characteristic of Scrum-of-Scrums is that often they become reactive in nature, and are simply a place to discuss impediments. Lean guidance would suggest that if impediments exist, then the process must be improved. The product-coordination team is proactive and creates a structure for cross-team planning and visibility, which is critical for scaling and sustainability.

You can use Scrum without automated acceptance testing or up-front unit testing

Most people are not aware that Scrum, as Jeff Sutherland originally created it, included automated testing practices and other quality engineering practices. Unfortunately, these were removed so that Scrum would catch on more readily. We say unfortunately because without them the quality of your code will degrade and you will find it hard to change and dangerous to change as well. In addition, the process of getting good acceptance tests helps clarify the customer’s needs and therefore lowers the risks of both building the wrong and thing building things wrong.

It is interesting to note that the Lean principles “optimize the whole” and “build quality in” are almost always violated when automated testing is not included. Optimize the whole, in this situation, means that when you are building a product you need to consider the entire time span of both the coding and testing as well as its whole lifespan, not merely the coding phase. Getting out a quick release that will cost a lot to maintain is not a good practice. As Scrum teams mature, most are starting to realize that automated testing should be a part of Scrum.

Here are three essential references if you want to learn more about this:

• Emergent Design: The Evolutionary Nature of the Software Profession (Bain 2008)

• eXtreme Programming Explained, Second Edition (Beck and Andres 2004)

• Working Effectively with Legacy Code (Feathers 2004)

Unfortunate Consequences of These Beliefs

These beliefs compound to form additional challenges.

Management should not prod teams for information if the teams decide they don’t need to give it

This sometimes arises from a combination of the fallacy that self-organized teams don’t need management’s help (they do!) and that teams need to be protected from management. The visual controls used by most teams should provide information to both the team and to management. Management has a need and a right to understand what is happening in the team. If it is difficult for a team to show this, there is something wrong with how the team is tracking their work. Lean’s use of visual controls provides guidance here.

No part of a team’s process can be dictated from the outside

It is unfortunate that many people still have the attitude that a central process group can find the right process for all teams. This is a holdover from a legacy mentality that does not work. Teams need to be responsible for their own process. However, Lean’s principle of optimizing the whole does mean that certain standards need to be established for how teams work together. Typically this is better accomplished with a “here is what needs to be achieved” mandate and not a “here is how to do it” one. Requiring teams to use consistent methods when they work together is a good practice—as long as each team can determine how best to meet this requirement.

Lean Thinking Provides the Necessary Foundation

Leaving teams to figure out for themselves their practices, workflows, and approaches almost guarantees misunderstandings, errors, and limitations; at best it is inefficient. Teams that guess well will succeed because Scrum is a useful approach. Teams that do not guess well may fail. Certainly, that is inefficient and results in a lot of unlearning, relearning, and adjusting.

The most effective method is to approach Scrum within the system of Lean thinking. Lean offers a well-established model to guide a team in its practices and in its workflow. Table 5.1 illustrates what help Lean thinking provides.

Table 5.1. Scrum and Lean Perspectives

Introducing Scrum#—Scrum Embedded in Lean Thinking

From Table 5.1, it is reasonable to expect that a team operating from the perspective on the left would get significantly different results—even with the same practices—than one using the explicitly stated Lean perspective on the right. Since both are following Scrum, we have chosen to label this second type—practicing Scrum within Lean’s context and belief system—as Scrum#. Scrum# is Scrum infused with Lean thinking.

It is useful to take this a little deeper than simply the beliefs. Using Lean thinking, what approaches should Scrum# practitioners follow?

At the beginning of any Agile transition, it’s best to avoid making too many changes at once; but starting with the four in Table 5.2 virtually guarantees better results. The quality of the team’s work will improve and cycle time will decrease.

Table 5.2 Four Essential Practices for Scrum#

Introducing Kanban Software Engineering

This section introduces Kanban software engineering,⁷ a relatively new approach to developing software rooted in Lean thinking. Based on long experience and good principles, many development teams see it as a healthier, Leaner alternative.

7. “Kanban software engineering” is perhaps an unfortunate name because it conjures images of the kanban cards that Toyota uses to manage their pull-manufacturing systems. Kanban software engineering is much more than merely using cards to manage WIP.

As Table 5.1 shows, most Agile methods use time-boxing, that is, managing software development by discovering and building in relatively small iterations. This indirectly improves workflow because the team works on small things and gets quick feedback to ensure it is working on the right things. Kanban software engineering focuses more directly on workflow.

Kanban software engineering (referred to as Kanban from now on) is based on the following beliefs:⁸

8. Most of the ideas regarding Kanban in this chapter come from Ladas 2009 and Anderson 2009.

• Software development is about creating and managing knowledge.

• Software development processes can be described in terms of queues and control loops, and managed accordingly.⁹

9. Ladas 2009, page 10.

• As information flows through the system, we must have some representation of it.¹⁰

10. Ibid., page 26.

The Kanban model, illustrated in Figure 5.1, is based on the notion that the team works on the appropriate number of features through completion. When the team is ready to begin on the next feature, they pull a feature from a small queue of potential work. This allows for proper management of both selecting what to work on and how to do the work.

• It focuses the team on building features that are as small as possible and that add value to the customer.

• The development pipeline has small queues and batches and so is more efficient.

• The team still gets quick feedback to keep them on track.

The differences between Kanban and common Agile approaches include

• The queues in front of software development teams stay small.

• The software development teams focus on completing features as quickly as possible but are not constrained by a time-boxed system.

• Kanban is explicit about including the entire value stream, from concept to consumption. Ideas from the customer start the value stream, and the product managers are directly tied to the teams because of the work-in-process limits that Kanban puts on this flow.

• No estimation is required in Kanban.

Figure 5.1 Flow of Kanban software engineering

Managing the Work in the Kanban Team

Kanban does not specify a technique for managing how work is done: It can be done individually or by a team swarm. Instead, Kanban seeks to control the amount of WIP that is allowed. Kanban accomplishes this by specifying slots for each available type of activity. Simply by limiting the number of slots available, we can limit the amount of WIP the team has at any step. By defining WIP limits for each activity, we can minimize the average cycle time for any activity.

The Kanban board (illustrated in Figure 5.2) helps the team manage its work. As team members complete a task, they move the card representing that work to the next step in the workflow. At any point in time, the board represents the current state of work. It also shows the process that the team is using and its WIP limits. The Kanban board could be considered a perfect “visual control” (discussed in chapter 8, Visual Controls for Enterprise Teams) because it accurately shows both process and status with minimal effort.

Figure 5.2 A Kanban board limiting WIP

Kanban’s approach—based on Lean thinking—is inclusive of management. This means that management is included in the conversations about how the work is being performed and tracked. This is important because it also means that management cannot just say “do more!” Instead, they agree to abide by the methods the team has selected to do their work. Chapter 11, Management’s Role in Lean-Agile discusses how managers lead and coach teams. By creating visibility into the team’s process (transparency), management can work with the team on improving that process.

Another diagram used by Kanban is the cumulative flow diagram (CFD), which describes the overall flow through the Kanban system; it provides a measurement for every significant step in the workflow. Figure 5.3 shows an idealized case with four steps: backlog (to be done), analysis, implement, and done. For each step, it shows the count of features at the given time interval. Wide lines indicate an impediment or blockage of flow while thin lines indicate that WIP is too small (sometimes called an “air bubble”).

Figure 5.3 Cumulative Flow Diagram

Advantages of Kanban

Many Agile teams spend ten to twenty percent of their time breaking down features into stories and estimating them. Sometimes this can be valuable for improving their understanding of the stories; however, when the story breakdown is required simply to make stories fit into an artificial deadline set by a time-boxed iteration scheme, it is wasted work. Compound this with the cost of estimating these smaller stories and it is a whole lot of expense and work for no extra value. Kanban eliminates this type of waste by managing for flow rather than for time boxes.

Kanban doesn’t assume all estimation is unnecessary, but it suggests looking at value received for time invested. Kanban pulls well from the features identified by the portfolio team. If you are fairly certain of your feature estimates, you may discover that detailed story estimates are not necessary.

The true value of Kanban lies in its requirement that the team create a workflow with explicitly defined rules and limits. This enables team members to discuss objectively what is working and what is not. That is, it helps the team focus on the process rather than on blaming a person. Yes, a person might have made a mistake, but what is it about the process that allowed the mistake to happen or to go undetected? Fix the process.

Think of Kanban this way; it combines

• Defining a workflow based on queues and control loops.

and

• Managing this workflow by limiting the amount of WIP at any step in the workflow.

Evidence suggests that teams learn continuous process improvement faster with Kanban. Some of the reasons are as follows:¹¹

11. Thanks to John Heintz for several of these insights.

• Kanban reduces the fear of committing to a per-story estimate, which is a significant risk for some teams. Fear always impedes learning.

• Kanban is explicitly a team process rather than one for individuals. It highlights the team’s performance rather than individuals’ and can reduce the fear of embarrassment.

• Kanban focuses on how the workflow process can be improved rather than blaming an individual.

• Kanban allows reflection about concrete measures such as, “Should WIP be 4 or 5?” Reflection about concrete issues is often easier in the beginning than reflecting about more abstract or personal issues.

• A transparent process allows management to be involved in improving it.

Now, we could take this conversation either way. Perhaps the PMs resolve it, perhaps they go to Joe. Maybe Joe realizes his mistake or decides the impact is too great or stays stubborn and can outrank everyone else. The point is the conversation is elevated to where it should be—at the product-management level.

Compare this with the typical situation in Scrum, which deals with conflict primarily at the level of a single product manager and the team. The problem is not between the product manager and the team, it is that the product managers aren’t prioritizing among themselves.

By creating transparency into the process, the demand’s impact is seen across the organization. The Kanban team doesn’t have to take a strong and uncomfortable stand, they merely let the business side see the impact their decisions will have on the productivity of the organization. Because the software team is not a black box to management, management can work with the team more effectively.

Selecting an Approach

Many choices, which one to use? Use the one that that fits the needs of your own particular context. There is not necessarily a right or wrong answer. Learn as much as you can and then consider the various tradeoffs. Table 5.3 compares how XP, Scrum, Kanban, and general Lean thinking address different factors.¹² Use this to help you select an approach for your team.

12. We believe Waterfall is virtually never appropriate if you have Kanban and Lean alternatives. Whereas XP and Scrum may not work in certain situations, with Lean and Kanban you can always use the underlying principles to figure out what to do.

Table 5.3 Selecting an Approach

Summary

While Scrum is an effective Agile project-management framework, its history, and the common misinterpretations of it, tend to limit its effectiveness when you want to extend it beyond a few teams. It never pays to be dogmatic in following Scrum. Scrum is meant to be a framework for creating an effective process for Agile development. One of its great mandates is to find impediments and remove them. While Scrum tells you not to follow an ineffective practice, it often doesn’t tell you what to do when its practices don’t seem to apply.

It is much more effective for teams to begin with approaches that are known to be good than to have to start from scratch. There is always more to learn; but you start out ahead when you can learn from the mistakes and successes of others. One effective approach is to let Lean thinking guide the practices of Scrum, to “embed Scrum in Lean thinking.” We call this approach Scrum#, and offer some specific practices that this approach prescribes.

Kanban software engineering is an emerging approach to software development that is also based on Lean thinking. Kanban seeks to improve the flow of products through the value stream by managing WIP directly. This is often a better approach than trying to manage flow through short iterations.

Wise development teams will use the approach—Scrum# or Kanban—that best fits their context.

Try This

These exercises are best done as a conversation with someone in your organization. After each exercise, ask each other if there are any actions either of you can take to improve your situation.

• If you have examples of failed Scrum projects in your organization:

• What was the reason(s) for failure?

• How did the organization react?

• Did anyone observe any of the misunderstandings presented in this chapter?

• If you are planning to use Scrum for the first time, are any of the misunderstandings we’ve talked about cited as justification for resistance?

• Review Table 5.3. What approach is best for your situation?

• How much WIP do you currently tolerate?

• How should you organize your storyboard so that it helps

• Control your WIP

• Management quickly grasp what they need to know

• Make visible and address resource constraints

• Manage dependencies in a visible way

Recommended Reading

The following works offer helpful insights into the topics of this chapter.

Alexander. 1979. The Timeless Way of Building. New York: Oxford University Press.

Anderson. June 8, 2009. Agile Management Blog: Thoughts on Software, Management, Constraints and Agility. www.agilemanagement.net/Articles/Weblog/KanbanBlogosphereRoundupJ.html.

Bain. 2008. Emergent Design: The Evolutionary Nature of Professional Software Development. Boston: Addison-Wesley.

Beck and Andres. 2004. Extreme Programming Explained: Embrace Change 2d ed. Boston: Addison-Wesley.

Denne and Cleland-Huang. 2003. Software by Numbers: Low-Risk, High-Return Development. Upper Saddle River, NJ: Prentice Hall.

Feathers. 2004. Working Effectively with Legacy Code. Upper Saddle River, NJ: Prentice Hall.

Gamma et al. 1994. Design Patterns: Elements of Reusable Object-Oriented Software. Boston: Addison-Wesley.

Kennedy. 2003. Product Development for the Lean Enterprise: Why Toyota’s System Is Four Times More Productive and How You Can Implement It. Richmond, VA: Oaklea Press.

Ladas. 2009. Scrumban: Essays on Kanban Systems for Lean Software Development. Seattle, WA: Modus Cooperandi Press.

Poppendieck and Poppendieck. 2006. Implementing Lean Software Development: From Concept to Cash. Boston: Addison-Wesley.

Reinertsen. 1997. Managing the Design Factory. New York: Free Press.

Shalloway and Trott. 2004. Design Patterns Explained: A New Perspective on Object-Oriented Design. Boston: Addison-Wesley.

Shalloway and Trott. 2009. The Lean-Agile Pocket Guide for Scrum Teams. Seattle: Net Objectives Press.

Womack and Jones. 2003. Lean Thinking: Banish Waste and Create Wealth in Your Corporation. 2d ed. New York: Simon & Schuster.

Womack and Shook. 2006. “Lean Management and the Role of Lean Leadership Webinar.” Lean Enterprise Institute. www.lean.org/Events/LeanManagementWebinar.cfm (accessed October 23, 2007).