Chapter 17. Scaling Scrum

My wife, Laura, cooks dinner nearly every night. Some nights she makes something a bit fancier; other nights, if she’s more rushed, she cooks something simple. But it’s always tasty, healthful, and prepared without a great deal of stress. Except for Christmas dinner. Cooking Christmas dinner is stressful. The house is full of guests—her parents, my parents, maybe an aunt and uncle, and a brother or sister or two. And yet she seems to prepare more dishes than we have guests. Christmas dinner is done at a scale unseen the rest of the year. And anything done at a larger scale than we are accustomed to—including a software development project—is more difficult.

When a software project gets large, it is complicated by more than just having more mouths to feed. Large projects are often more critical to the organization, under greater scrutiny, more time-sensitive, more prone to personality clashes, longer, and more likely to be distributed across multiple sites.

The first round of defense against large projects is to attack them not with one large team but with multiple small teams. Chapter 10, “Team Structure,” introduced the idea of the two-pizza team, a team of perhaps five to nine members—small enough that it can be fed with two pizzas. When faced with a large project, we will use many of these two-pizza development teams rather than one much larger team.

In this chapter we look at ways to overcome the challenge of successfully using Scrum on a large, multiteam project. Specifically, we look at scaling the product owner role, working with a large product backlog, managing dependencies among teams, coordinating work among teams, scaling the sprint planning meeting, and the role of communities of practice on large Scrum projects.

Scaling the Product Owner

The product owner role can be one of the most challenging on a Scrum project. On all projects, the product owner is torn between competing inward-facing and outward-facing needs. Among the inward-facing tasks are participating in planning meetings, sprint reviews, sprint retrospectives, and daily scrums; managing the product backlog; answering questions from the team; and simply being available to the team during the sprint. A product owner’s outward-facing tasks include talking to users about their needs, creating and interpreting user surveys, traveling to customer sites, attending industry trade shows, managing stakeholder expectations, prioritizing the product backlog, determining product pricing, developing a medium- and long-term product strategy, watching for industry and market trends, performing competitive analysis, and more. On a project with one team of developers, this is often a vast but achievable amount of work. On a large project with multiple teams, however, the product owner role is too big for one person, so we must find ways of scaling it.

As a project grows to include multiple teams, ideally a new product owner is found for each. If you cannot achieve a one-to-one correspondence between teams and product owners, try to have each product owner responsible for no more than two teams. This is usually the most that one product owner can effectively work with.

At some point as the overall size of the project grows, it makes sense to introduce a hierarchy of collaborating product owners. Figure 17.1 shows such a hierarchy, with a product owner working with each team, two product line owners each working with a cluster of teams, and a chief product owner. Naturally, layers can be added or removed as needed for the scale of the project.

Figure 17.1 The product owner role can scale up to include product line owners and a chief product owner.

Sharing Responsibility, Dividing Functionality

The chief product owner is responsible for having an overall vision of the entire product or product suite. The chief product owner conveys this to the entire team in whole-team meetings, e-mails, team get-togethers, and through whatever other means are available. But the chief product owner is almost certainly too busy to assume hands-on responsibility for working as the actual product owner for one of the five- to nine-person teams building the product. At this level, the external-facing requirements of the role are too great. A good chief product owner will be very involved with teams—attending daily scrums occasionally, walking through team areas whenever in the office, and offering support and feedback. But the chief product owner will need to rely on the product line owners and product owners to handle the intricate details of their product segments within the overall project vision.

Suppose, for example, we decide to develop an office productivity suite that will include a word processor, spreadsheet, presentation software, and personal database. Competing with Microsoft Office, Google Apps, and other products will be daunting, but our chief product owner is fearless. Because the chief product owner will be focused on strategic issues, competitive positioning, and such, product line owners are selected to own the individual products in the suite—the word processor, spreadsheet, presentation program, and database. Each product line owner in turn identifies product owners who will be responsible for feature areas within the product. The product line owner for the word processor, for example, may work with one product owner who is responsible for tables, another responsible for stylesheets and printing, another who is responsible for the spell checker, and so on.

Although as previously mentioned, the chief product owner is too busy to be the product owner for one team, it is possible for a chief product owner to act as the product line owner for part of the product. Continuing with the preceding example, our chief product owner may choose to also be the product line owner responsible for the word processor, perhaps because of being in that role previously. Similarly, a product line owner will often want to stay involved in a more hands-on manner and will work also as one of the product owners. Perhaps our product line owner for the spreadsheet product also acts as the product owner for the team that will add charts to the spreadsheet product.

Although functionality can be divided along these lines, it is important for all product owners to feel a shared responsibility for the full product. They must also instill this feeling of shared responsibility in the teams they work with.

Working with a Large Product Backlog

Most large project teams opt to use one of the commercial agile tools that provide support for working with a large product backlog. I won’t, therefore, go into all of my preferences for how to work with a single large product backlog because much of how an organization works with its product backlog will be dictated by its tool selection. There are, however, two guidelines worth pointing out that remain valid regardless of which backlog management tool is selected:

• If there’s only one product, there should be only one product backlog.

• The product backlog should be kept to a reasonable size.

These topics are addressed in the sections that follow.

One Product, One Product Backlog

There’s a reason it’s not called a “project backlog” or a “team backlog” or any other similar but deficient term. It’s called a product backlog because there should be one per product. If a team is working from more than one product backlog, the multiple backlogs must be prioritized against one another. It is not enough to prioritize each product backlog and tell the team to take the top five items off of each. The top item on one product backlog might be lower priority than the lowest item on another product backlog.

As an example, consider successful Scrum adopter Ultimate Software of Weston, Florida. Ultimate Software develops Software as a Service (SaaS) for human capital management. This includes features for human resources and payroll management. Although these features are obviously entwined (the human resources you manage need to be paid), the underlying software is modular. Within Ultimate there are teams that focus on enhancing the human resources portion of the product and other teams that focus on enhancing the payroll portion. However, even with teams focused on different areas of the system, Ultimate maintains a single product backlog for the overall product.

Having a single product backlog allows the chief product owner at Ultimate to see the relationship between the top-priority human resources features and the top-priority payroll features. Suppose that all of the items at the top of the product backlog are human resources features. This is an indicator to the chief product owner to either redirect the payroll-focused teams toward human resources features (where they’ll be less productive at first because they aren’t as familiar with that domain or code) or have them continue on lower-priority payroll features.

Logistical problems can arise, however, when multiple teams and multiple product owners work with a single product backlog. Although we can agree that all features should be prioritized relative to all other features, this can be extremely difficult to do on a project with numerous product owners, multiple product line owners, and a chief product owner. Rather than allowing each product owner to maintain his or her own private product backlog, a better solution is to have a single product backlog but to provide views into it for each product owner. This can be seen in Figure 17.2.

Figure 17.2 There should be one product backlog per product, but there can be multiple views into it.

Figure 17.2 shows two teams sharing a common product backlog. When the team on the right, which is developing the human resources features of the system, views the product backlog, it sees items that the team is or could be responsible for delivering. The payroll team on the left has a similar view of the product backlog, which shows items of interest to that team. Notice that some product backlog items may be shown in both views. This can indicate that the feature may be fully implemented by either team or that both teams will need to be involved in developing it. Suppose, for example that the word processor and spreadsheet teams of an office suite product were looking at their product backlog views. It is likely that each would see a product backlog item for enhancing the shared spell checker.

Keep the Product Backlog to a Reasonable Size

We need to balance our desire for a single product backlog with the competing desire that the product backlog not become unmanageable. In fact, from my experience, things degrade rapidly if anyone involved in the project is expected to be familiar with more than 100–150 items. I have two reasons why I think this to be a reasonable upper limit. First, by observing and working with hundreds of Scrum teams, when I hear a complaint of “our product backlog is too big,” it is almost always in reference to a product backlog with 100 or more items. My second argument in support of keeping the product backlog under 100–150 has to do with a release party I went to in 2000.

As the vice president of development in this organization, I was pleased when my teams finished a particularly large project on time. We decided to have a party to congratulate the team and celebrate its success. The party was at a hotel banquet room and was attended by 160 team members and their significant others. I walked around the room with my wife, introducing her to those I worked with and meeting their guests. Until, uh-oh, I saw a couple and had no idea which of the two I worked with. This was bad—one of them reported to me or to someone who reported to me, but I couldn’t remember if it was him or her. My brain failed me because of something called Dunbar’s Number, which I read about for the first time shortly after this party.

Robin Dunbar was a British anthropologist who suggested that the human brain has a limit of around 150 individuals with whom one can maintain a normal social relationship. Up to approximately 150 people, and we can remember who each person is and how each relates to all others (“That’s Joachim; he’s in the testing group”). Beyond that number, we get confused. Aha! This explained why I couldn’t remember each person at the party. But what does this have to do with product backlogs? Perhaps not much at a scientific level. However, if the human brain is wired such that we can only remember 150 people and the relationships among them, I’m willing to go along with the idea that most of us can only keep 100–150 or so product backlog items and the relationships among them straight in our brains. Combined with anecdotal evidence that teams with 100 or more items on their product backlogs complain about them to me and teams with smaller product backlogs don’t, 100–150 seems like a good upper limit.

Although this may seem to be a small number of product backlog items, consider that we have two ways of keeping the number of backlog items manageable.

• Make use of epics and themes. By writing some large stories (epics) on the product backlog and by grouping small stories together into themes, even the product backlog for an immense project can be made to fit within my guideline of no more than approximately 150 product backlog items.

• Provide views into the product backlog. Notice that I didn’t say there cannot be more than 150 items on the product backlog; I only said that no one should need to be familiar with more than that many. This can be achieved by providing multiple views into the product backlog, as shown earlier in Figure 17.2. Imagine again that you are building a competitor to Microsoft Office and Google Apps. The chief product owner of such a product would be overwhelmed if she had to be aware of and look at 500 or more product backlog items at one time. It would be more useful for a chief product owner such as this to be able to see rollups of individual product backlog items into themes while allowing the individual user stories to be visible to teams.

This can be seen in Figure 17.3, which shows a partial product backlog. The chief product owner can see themes, which are groups of related user stories. The individual teams and their product owners, however, can see individual user stories written at the level of detail they need to turn each into new capabilities in the product.

Figure 17.3 Multiple views into one product backlog.

Proactively Manage Dependencies

For the most part, Tom’s project was moving along fairly well: The teams had become accustomed to the iterative and incremental nature of Scrum and were beginning to truly embrace automated testing, test-driven development, and even pair programming. Tom had embraced his new role as ScrumMaster with only a few remaining traces of his previous life as a command-and-control project manager. After every sprint, Tom’s two teams would demonstrate the new functionality in their online payment and money transfer application. The stakeholders were pleased at the progress. Usually. Every now and then, Tom’s teams would complete only a small portion of what they’d committed to. The teams used their retrospectives to consider root causes each time this happened. In an e-mail to me about the situation, one of the team members, Campbell, summarized what they learned.

Almost every time a sprint blew up on us, it was because of a dependency between our two teams. We wouldn’t think about it during sprint planning. Or we’d think about it but misjudge the effort involved.

On any multiple team project such as Tom’s and Campbell’s, the potential for dependencies between teams exists. Good team structure can go a long way toward reducing dependencies but will not eliminate them. Similarly, continuous integration helps point out problems caused by some dependencies. Fortunately, there are additional techniques Scrum teams can employ to further manage dependencies. These include doing rolling lookahead planning, holding release kickoff meetings, sharing team members, and even using a dedicated integration team.

Do Rolling Lookahead Planning

All too often a team finishes its sprint planning meeting only to discover it needs a small amount of work done by another team but that team is not available. Rolling lookahead planning greatly reduces the frequency of this problem by having teams spend a few minutes each sprint thinking about what they will do in the next couple of sprints. Normally the most convenient time to do this is at the end of the sprint planning meeting, as the team and product owner are already together with their minds on planning.

Depending on the team, the sprint length, and a few other factors, planning one sprint takes anywhere from an hour to a full day. Planning an additional couple of sprints in the same meeting would seem both impossible and tediously boring. Fortunately, we only need to look ahead at the subsequent two sprints using average historical velocity, with no consideration of tasks or hours. This means planning those two sprints will take about ten minutes, assuming the team knows its historical average velocity and the product owner has prioritized the product backlog.

So, a team doing rolling lookahead planning will walk out of the meeting with the coming sprint planned in detail (a selected set of product backlog items and the tasks with hours for each) and a set of product backlog items tentatively selected for the following two sprints. An example can be seen in Table 17.1, which illustrates what a rolling lookahead plan might look like after the sprint planning for sprint three. One sprint later, the team will identify tasks for the product backlog items of sprint four and will peer forward at sprints five and six, which will have rolled into view. It is worth noting that the product backlog items selected for the later sprints are subject to change when work begins on those sprints. Rolling lookahead planning should be viewed as a chance to consider what might be worked on next, not as a locked-down plan. The product owner will still be able to revise his or her opinion based on then-current information.

Table 17.1 Rolling lookahead plans include details for the current sprint but only the high-level items of the next two sprints.

I recommend looking two sprints ahead because doing so gives teams adequate time to respond to most newly discovered dependencies. To see why, consider what would happen if a team were to plan only one sprint ahead. If you need something from another team by the start of the next sprint, your only option is to ask that team to do it in the coming sprint. If that team were planning the coming sprint at the same time you were, it has probably fully planned its sprint by the time you ask. Looking ahead two sprints, though, allows you to go to that team with more advance notice. The other team has time to plan and perform the needed work in the next sprint and have it ready for you by the start of the following sprint, which is when you needed it. Some teams—usually those involved in hardware or embedded development—may look even further ahead with rolling lookahead planning.

Hold a Release Kickoff Meeting

Another technique for proactively managing dependencies is to bring everyone together for a release kickoff meeting. An ideal time for doing so is the start of a new project or release cycle. A kickoff meeting can mitigate one of the biggest risks on a large project: that the different teams or individuals will start pulling in wrong or different directions.

Prior to a release kickoff meeting, each two-pizza team and its product owner has created a rough plan of what will be delivered over a reasonably foreseeable period, typically around three months. At the release kickoff, these plans are shared with everyone involved in the project. Normally, the product owners for each team take turns sharing what their teams plan to work on.

Salesforce.com delivers a new release of its SaaS platform every three or four months and has found release kickoff meetings essential. Eric Babinet and Rajani Ramanathan say that “it is difficult to identify dependencies or negotiate commitments with other teams if those teams don’t yet know what they’re doing in the release. The Release Kickoff meeting acts as an important synchronization point for teams and helps ensure more productive discussions around inter-team dependencies” (2008, 403).

Teams at Salesforce.com have introduced a worthwhile innovation beyond a standard release kickoff meeting. Later in the same week they hold what they call a “Release Open Space.” This is patterned after the Open Space Technology approach¹ that has become popular at conferences in recent years. Each team is asked to send at least one person to the open space. This informal meeting starts with individuals identifying topics of interest relevant to the release and writing them on large pieces of paper taped to the walls. After topics have been identified, groups form to discuss the topics of interest. Salesforce.com provides 45 minutes for discussion followed by a 30-minute debrief with everyone back together. The cycle is repeated as long as there is passionate interest in the topics remaining on the walls.

1 Open space is a self-organizing approach to meetings, conferences, and the like. It has become a staple of agile conferences. For more information see http://en.wikipedia.org/wiki/Open_Space_Technology.

Share Team Members

Another possible approach for proactively managing dependencies is to share team members among teams. This is an effective approach when dependencies are difficult to identify in advance or when they need to be addressed quickly. This is not a very effective strategy when dependencies can occur among any number of teams and in any direction. However, it is a good strategy when dependencies are likely to exist between feature teams and component teams.

Using this approach, the shared team member is on two teams concurrently, working on both sides of a known or likely dependency. This can be seen in Figure 17.4, which shows three feature teams, two of which share a part-time member with a component team. Additionally, two of the feature teams share a member.

Figure 17.4 Sharing a few individuals between teams is a good way to ensure interteam communication.

Use an Integration Team

Although sharing a part-time team member is sometimes a step in the right direction, it may not be a step far enough. Sometimes it is necessary to create an integration team. This is most common on projects with ten or more teams. An integration team works in the gaps that may exist between the development teams. Most of these gaps occur in the interfaces between teams (Sosa, Eppinger, and Rowles 2007). Interface problems can be broadly split into these two categories:

• Unidentified interfaces. An unidentified interface is one that exists but that no one has discovered yet.

• Unattended interfaces. An unattended interface is one that exists and that at least one team is aware of but which no one is doing anything about.

Integration teams focus directly on unattended interfaces, while being on the lookout for unidentified interfaces. After an integration team has uncovered an unattended or unidentified interface, its first strategy should be to encourage one of the development teams to assume responsibility for it. When that is impossible or impractical, the integration team takes ownership of the interface.

Normally, the first thing integration team members do each morning is check the results of the official nightly build to make sure that the system built successfully and that all tests passed. If it did not, integration team members do whatever is necessary to get all tests to pass. This will usually involve identifying the problem, finding the source of the problem, determining which team or teams are involved, and then working with those teams to resolve the issue.

On a large project, an integration team may be made up of full-time team members who work only on the integration team. In fact, on a very large project, there may be multiple integration teams, each with full-time members. Many other projects—those with perhaps a handful to a dozen teams—get along quite well with a virtual integration team, where individuals are assigned to the integration team but remain primarily on their individual development teams. Members of a virtual integration team meet each morning to assess the state of the previous night’s build and agree upon who will resolve any issues. Members then spend the rest of the day working with their individual development teams.

It is also common at the start of a new project to dedicate individuals to an integration team for a few sprints. This team is tasked with getting all of the necessary servers installed and configuring project-wide software such as wikis, continuous integration servers, and so on. After these systems are in place, integration team members return to their development teams, and the integration team becomes a part-time endeavor.

As an example, consider a large San Francisco-based bioinformatics company that has a dozen feature teams, two component teams, and one integration team. Beyond monitoring the nightly build and fixing issues with it, the integration team develops automated tests to verify integration points. These are the types of tests that aren’t obviously the responsibility of either team but that someone needs to be responsible for before the product can ship. Because these are often unidentified interfaces, integration team members spend a lot of effort looking for potential trouble spots. For example, there is usually one representative from an integration team at all of the standard meetings held by the feature and component teams. It is not unheard of for a member of an integration team to attend three daily scrums some days. They are at those meetings—plus sprint planning, review, and retrospective meetings—listening for unattended and unidentified interfaces.

Because being on an integration team requires good analytical skills, including the ability to connect comments made weeks apart by different teams, you should be careful not to consider an integration team as a dumping ground for poor performers. Integration teams require senior people with broad skills. That being said, an initial tour of duty on an integration team is an excellent way to provide new employees with a broad view of the overall system. It also allows new employees a very structured way to meet just about everyone else on the project and to form important connections that will serve them well later. Just make sure that your integration team isn’t mostly made up of new employees.

Coordinate Work Among Teams

Because Scrum scales by having multiple small teams rather than one large team, the problem arises of how to coordinate the work of all those teams. A ScrumMaster named Joanne told me how she learned the importance of this after running her first multiple-team project. Fresh from success on a one-team Scrum pilot, Joanne accepted the challenge of being the ScrumMaster for five teams working together to deliver a new version of her company’s ambulance dispatch product. She provided short training to each team and turned them loose. Things went as well as could be expected for the first four sprints until the dependencies among teams became more critical. It then became apparent that the teams were working in isolation, each trying to go as fast as possible toward its own goals but failing to pay adequate attention to integration points. Joanne admitted to me in an e-mail that she did too little to encourage cross-team communication.

Everyone was very good at figuring out what needed to be done on his own team. If we forgot something during sprint planning, the team would be all over it during the sprint and resolve the issue. But no one was watching for the thousand little issues that were piling up between teams. It was like two baseball players who watch the ball fall between them because each thinks the other will catch it.

There was no animosity or competitiveness among teams. It was merely that each team was so focused on what it considered its own goals that it ignored the overall goal. Chapter 11, “Teamwork,” introduced the idea that Scrum teams are built on whole-team thinking and shared responsibility. On multi-team projects, the “whole team” is not just one two-pizza team along with its product owner and ScrumMaster, but all of the two-pizza teams, product owners, and ScrumMasters together.

In this section we will look at what Joanne could have done to improve the coordination of effort among her teams. Specifically we will look at look how to conduct scrum of scrums meetings and whether teams should synchronize sprint start and end dates.

The Scrum of Scrums Meeting

A fairly universal practice for coordinating work among several teams is the scrum of scrums meeting. These meetings allow clusters of teams to discuss their work, focusing especially on areas of overlap and integration.

Imagine a perfectly balanced project comprising seven teams each with seven team members. Each of the seven teams would independently conduct its own daily scrum. Each team would then also designate one person to attend a scrum of scrums meeting. The decision of who to send should belong to the team. Usually the person chosen should be a technical contributor on the team—a programmer, tester, database administrator, or designer, for example—rather than a ScrumMaster or product owner. Being chosen to attend the scrum of scrums meeting is not a life sentence. The attendees will change over the course of a typical project. The team should choose its representative based on who will be in the best position to understand and comment on the issues most likely to arise at that time during a project.

If the number of teams participating is small, it may be acceptable for each team to send two representatives—a technical contributor, as previously described, and the team’s ScrumMaster—if the teams desire. I tend to do this only when there are four or fewer teams, which keeps the meeting size to eight or less. Most scrum of scrums groups do not designate a specific ScrumMaster for themselves. After all, the individuals who attend are accustomed to participation on self-organizing teams. In some groups, however, someone volunteers to act as the ScrumMaster or just assumes the role. It is up to the group to decide if this is acceptable.

The scrum of scrums meetings can be scaled up in a recursive manner. If a large product is being built by many teams of teams, one representative of each scrum of scrums can attend what might be called a “scrum of scrum of scrums” meeting, although this starts to sound silly, and most organizations stick to calling it a scrum of scrums meeting regardless of the level at which it is occurring. An example of this can be seen in Figure 17.5, which shows 11 individual teams. The 11 teams are part of 3 teams of teams, each of which has its own meeting. But since those 3 teams of teams combine their work into one product, there is another level of meeting, attended by one person from each of the scrum of scrums meetings.

Figure 17.5 Scrum of scrums meetings can be applied recursively at as many layers as needed to coordinate work among clusters of teams.

Frequency

Scrum of scrums meetings differ from daily scrums in three important ways:

• They do not need to be held daily.

• They do not need to be timeboxed to 15 minutes.

• They are problem-solving meetings.

I find that holding scrum of scrums meetings two or three times a week is sufficient for most projects. This makes a Tuesday-Thursday or Monday-Wednesday-Friday schedule appropriate. Although a scrum of scrums meeting will often be completed in 15 minutes, I recommend blocking 30 or 60 minutes for them on the calendar. This is because, unlike daily scrums, these are problem-solving meetings. If an issue is brought to the attention of this group, and the right people are present to address the issue, they should do so.

Think about how many people might be waiting for a resolution. There could be close to 100 people waiting for an answer from a scrum of scrums meeting (and many more from a scrum of scrum of scrums meeting). Issues brought to this group should be resolved as quickly as possible, which means the meetings cannot be as readily timeboxed and issues cannot just be left for another day.

Sometimes, of course, issues arise that cannot be immediately addressed. Perhaps other people are needed to address the issue or additional information is needed. When an issue cannot be resolved immediately it is placed on the group’s issues backlog, which is a list of outstanding issues that the scrum of scrums group either plans to resolve or wants to track to make sure another group resolves them. Often a simple, low-tech tracking mechanism is adequate for this backlog. Most teams use a large piece of paper hanging in a team room, a spreadsheet, or a wiki.

Agenda

A scrum of scrums meeting will feel nothing like a daily scrum despite the similarities in names. The daily scrum is a synchronization meeting: individual team members come together to communicate about their work and synchronize their efforts. The scrum of scrums, on the other hand, is a problem-solving meeting and will not have the same quick, get-in-get-out tone of a daily scrum. The agenda for the scrum of scrums meeting is shown in Table 17.2. As you can see from this table, the scrum of scrums, like the daily scrum, starts with each attendee answering three questions:

1. What has my team done since we last met that could affect other teams?

2. What will my team do before we meet again that could affect other teams?

3. What problems is my team having with which it could use help from other teams?

Table 17.2 An agenda for the scrum of scrums meeting includes three questions followed by a discussion of an issues backlog.

Topics raised during this discussion are added to the group’s issues backlog. This part of the meeting is meant to be fast-paced and fairly short. You should timebox it to 15 minutes as you would a daily scrum. One technique for achieving this is adopting a guideline that personal names should be avoided. There are two reasons for this. First, leaving out names keeps the discussion at the appropriate level of detail. While attending the meeting, I want to hear about each team, not about each person on each team. Second, too many people equate importance with how long they talk during a meeting. Following this guideline will keep this part of the meeting moving briskly.

After everyone has answered the three questions, the meeting participants address any issues, problems, or challenges that were raised during the initial discussion or that were already on the issues backlog.

Scrum of scrum groups do not conduct formal sprint planning and sprint reviews. Participants in these meetings are first and foremost individual contributors on their teams. The higher-level scrum of scrums is a more transient group, with members changing occasionally throughout the project. The sprint planning and commitments that most drive a project forward rightly belong at the individual team level.

Synchronize Sprints

On my first Scrum project, we started with only one team. That project soon grew to three teams, with the typical dependencies between them. I quickly arrived at what I thought would be a good way to manage those dependencies. I would stagger the sprint start dates by a week, as shown in Figure 17.6. The idea was that when a team went to start its sprint it would know the stories one of the other teams had recently committed to and which stories the other team was likely to finish.

Figure 17.6 Overlapping sprints cause problems.

Well, that part of my plan did work out well. But, overall, staggering the sprint start dates was a horrible idea. The biggest flaw in overlapping sprints is that there is never a time (except the end of the project) when all teams are done. One or more teams are always partway into a sprint. Some are planning a new sprint, others just planned a week ago, and still more teams will plan next week. This makes it difficult to give the full system to a customer for feedback or to an operations group for deployment.

All sprints do not necessarily need to end on exactly the same day. It is acceptable on a large project to have sprints that end over a two- or three-day period. In fact, there can be advantages to doing this. Allowing sprints to end over a two- or three-day period can make it easier for someone on multiple teams to attend all the review and planning meetings expected of someone on multiple teams. Additionally, it has the advantage in many cases of better accommodating remote team members who may travel into town for these meetings. A remote team member who is on multiple teams will find it easier to justify the travel time and expense if she can participate fully in each of her teams’ meetings.

Although the primary benefit of synchronized sprints is that all teams start and stop within a day or two of each other, this does not mean that all teams must work in sprints of the same length. A project with multiple teams can accommodate different sprint lengths through the use of nested sprints, as shown in Figure 17.7. The most common use of nested sprints is when the various teams on the project cannot agree on a common sprint length, with some wanting two-week sprints and others wanting four-week sprints.

Figure 17.7 For sprints to be synchronized, not all sprints need to be the same length.

Scaling the Sprint Planning Meeting

Most of the standard meetings held by Scrum teams are deliberately unaffected as the overall project size grows and multiple teams are involved. Teams continue to hold daily scrums, sprint reviews, and sprint retrospectives just as they would if working on a project as a lone team. Sure, teams will sometimes decide to hold joint reviews when it makes more sense to view the work of multiple teams together. And teams will occasionally hold joint retrospectives both to mix things up and to perhaps focus on interteam issues. But it is the sprint planning meeting that is most severely impacted when a Scrum project grows to include multiple teams.

With multiple teams working on the same project, a number of problems arise during the sprint planning meeting, including the following:

• Some people are needed in multiple sprint planning meetings. With all sprints sharing a common start date, people need to be in two places at once.

• If one team discovers a dependency on another team, it may not be able to get the other team to commit to taking on a task if the other team finishes its planning first.

• If multiple teams are pulling items from the same product backlog, those items have to be preallocated to teams before sprint planning starts.

Fortunately, there are a couple of approaches to large-scale sprint planning that can reduce or eliminate these problems and others like them.

Stagger by a Day

As described in the earlier section on synchronized sprints, the benefits of synchronizing can still be had even with sprints that are offset by a day or two. We can take advantage of this, and instead of having all teams do sprint planning on exactly the same day, we can have one-third of the teams plan on, say, Tuesday, one-third on Wednesday, and the final third on Thursday. When teams do this, the issues surrounding a shared product backlog are, fortunately, mostly eliminated. When my team plans tomorrow, we will know which product backlog items your team committed to today.

Staggering by a day also addresses the problem of requiring an in-demand product owner, architect, user interface designer, or other shared team member to be in two places at once. However, it replaces that problem with perhaps “three days of pain,” as one product owner put it. The shared team member is able to help more teams, but doing so can be exhausting; no one enjoys attending planning meetings for two or three full days straight.

Despite this drawback, staggering by a day remains a valid option, usually for projects of up to nine teams, with three teams planning each day. After that, even with staggering, there are usually too many days of planning, so other approaches are needed.

The Big Room

In the big-room approach, all of the teams (or as many as will fit) are assembled into a large room. The chief product owner starts things off with any remarks he or she would like to share with the group of teams: perhaps the results of recent discussions with customers, prospects, or users; or maybe a general description of the types of things that everyone will be working on for the coming sprint or two. After the meeting is kicked off, each team (including its product owner and ScrumMaster) moves to a portion of the room where it will be able to work together for a few hours. Some teams stake out the corners, others claim the side walls, a few more pull some tables together in the middle of the room. In these locations, each team works on planning its coming sprint in the same manner it would if it were the only team on the project.

The room gets loud and cacophonous, but the energy and hopefully the passion in the air are obvious. As teams plan their sprints, it won’t be long before they begin to uncover dependencies on each other. As this happens, one person jumps up, runs over to the other team’s temporary camp (which is in the same room), and asks, “Can your group do such-and-such for us this sprint? We’ll need it if we want to finish one of the backlog items our product owner wants us to do.” The team member can either wait for a reply or return to her team and wait for an answer to be shouted or ferried across the room a few minutes later.

The big-room approach also works exceptionally well for critical, shared resources. A product owner who is working with two teams can bounce between them; the company’s chief software architect, too busy to be on any one team but needed by all, can move about the room, going where called. On most projects a team usually signals its need for a product owner, architect, or other shared resource by yelling to the person. This works well, except by the time the shared resource finishes his current discussion, he forgets who called him.

A technique I’ve found successful is to augment yelling by using nautical signaling flags as shown in Figure 17.8. When team members need the architect, for example, they hang the appropriate 1′ × 1′ signal flag near their area. When the architect finishes what he’s doing, he can look around the room for his flag. If team members need a product owner, they would display a different flag. A team displaying a product owner flag would be helped by either its product owner or perhaps the product line owner or even chief product owner when possible. This works well: It solves most problems of knowing who needs help next, it lets shared resources see how many teams need help, and it is fun.

Figure 17.8 Flags can be used to signal for help—at sea or in a meeting.

For the big-room approach to work, the product owners must be prepared prior to the meeting. This usually entails a variety of short meetings between the chief product owner and his or her staff. Each level of the product owner hierarchy needs to understand the product vision as it trickles down from the chief product owner through the various levels.

Cultivate Communities of Practice

On a multi-team project, it is possible for individuals to become isolated, speaking mostly to others on their individual teams. Good ideas are slow to propagate across the organization. Similar functionality is implemented differently by different teams. We put scrum of scrums meetings in place to reduce the impact of some of these problems, but those only go so far. An additional solution and one that is critical to the success of any large Scrum project is the cultivation of communities of practice. Like improvement communities (a special type of community of practice introduced in Chapter 4, “Iterating Toward Agility”), a community of practice is a like-minded or like-skilled group of individuals who voluntarily come together because of their passion and commitment around a technology, approach, or vision. On a large project, these communities of practice are helpful for cutting across the boundaries of and pulling together individuals from the many cross-functional teams. An example can be seen in Figure 17.9.

Figure 17.9 Communities of practice cut across the development teams and create additional channels of communication.

Figure 17.9 shows communities of practice formed simply around various project roles. In addition, a sufficiently large project might have communities of practice form around technologies (a Ruby community and a .NET community, for example), around interests (mock objects, artificial intelligence, test automation), or around any thread of common interest to multiple development teams.

A good example of a community of practice is the Virtual Architecture Team at Salesforce.com, as described by Eric Babinet and Rajani Ramanathan.

The Virtual Architecture Team (VAT) is “virtual” as it is made up of developers from every Scrum team. Members work on the VAT in addition to being on a Scrum team. The VAT owns maintaining and extending our industry-leading software architecture. They do this by defining the architectural road map, by reviewing architecturally significant changes to the code, and by defining standards to ensure consistent and maintainable code. (2008, 405)

Communities of practice can span more than one project. For example, a community of practice around test automation may form and include members from multiple, completely unrelated projects. Unlike improvement communities, a general community of practice does not form around the pursuit of a single, specific goal. Instead, a community will generally have many related goals. Because of this, a community of practice can remain in existence indefinitely. A community can also dissolve after it has achieved its goals or if members lose passion.

Because they span teams, communities of practice are a primary mechanism for spreading good ideas among teams and for ensuring desirable levels of consistency or commonality across a set of development teams. A community of middle-tier programmers might, for example, discuss and decide when would be the best time to upgrade to the latest version of application server software for a family of products. Discussions among members of an orthogonal test team would ensure consistent test tool usage and the sharing of good practices.

Formal or Informal

A community of practice can be either formal or informal; most organizations will have a mix. Organizations with strong functional management in place prior to adopting Scrum usually rely on those strong functional managers to support or even establish some of the communities of practice. Etienne Wenger, who invented the term “community of practice,” and his colleagues distinguish five types of communities of practice based on the degree of recognition of the community by the organization. These are shown in Table 17.3.

Table 17.3 Communities of practice range from those no one knows exist to those given official status. (Adapted from Wenger, McDermott, and Snyder 2002.)

Table 17.3 is not meant to represent a hierarchy. No one type of community of practice is always superior to another; each has its unique strengths and weaknesses. Similarly, Table 17.3 is not meant to indicate the life cycle of a typical community of practice, even though it is easy to see how a community could move from unrecognized to institutionalized as its purpose becomes clearer to the community members and its value becomes clearer to the organization. Although communities of practice do sometimes follow this life cycle, it is not by design. Many communities will move in the opposite direction, and some will disband entirely.

Creating an Environment for Communities to Form and Flourish

The most effective type of community of practice within Scrum organizations seems to be one that forms organically rather than by management request, although both approaches can be used for different purposes. Because self-organization is critical to successful agile teamwork, the formation of self-organizing communities of practice creates a powerful synergy. In this sense it is up to the organization and its leadership to create an environment in which communities of practice can form, flourish, and then fade away as they run their course.

According to Etienne Wenger and his colleagues, “because communities of practice are organic, designing them is more a matter of shepherding their evolution than creating them from scratch” (Wenger, McDermott, and Snyder 2002, 51). They provide seven principles for creating an environment that supports this evolution:

• Design for evolution. Acknowledge that each community will change over time. It will rise and fall in importance, members will come and go, goals will shift. This is good as it represents the progress of each community adjusting in response to changing project, people, and organizational needs.

• Open a dialogue between inside and outside participants. Although communities work largely within themselves, they cannot work in isolation from the overall organization. A good community is receptive to hearing what the organization needs, is struggling with, and can provide.

• Invite different levels of participation. Not everyone interested in a community will have the same amount of time and energy to devote to it. Encourage individuals to participate at the level and frequency that suits them.

• Have both public and private events. A good community understands that sometimes the most beneficial discussions are among no one but themselves, but also understands that public events are sometimes necessary. An example might be a community focused on improving the experience across a suite of products. Members meet privately every other week to bounce ideas around but also occasionally host open sessions where anyone in the company can attend and see what they’re thinking.

• Focus on value. The more value a community of practice provides to the organization, the more additional communities will be encouraged to form. Also, as communities deliver value, they are given more support and leeway in how they operate.

• Combine familiarity with excitement. A mature community of practice often settles into a series of habits—the weekly conference calls, the monthly meeting, and the annual two-day get-together at headquarters, for example. Although there is value in this, there is also value in occasionally mixing things up, which creates a sense of vibrancy within the community. Inviting an outside speaker with a different opinion or holding an open space–style event are just two examples of how to add some excitement.

• Create a rhythm for the community. Communities of practice do not generally work in the same, regular sprints of Scrum development teams. Without the project-oriented deliverables of a development team, sprints are not a must. However, a community still benefits from the establishment of a regular rhythm. Accomplish this by having a regular set of activities at whatever frequency suits the community.

If you want communities of practice to form organically, you may need to provide some encouragement. Potential community members will need to know that it is OK—and in fact encouraged—for them to form communities. I’ve encountered a few situations where managers and executives gave their Scrum teams a great deal of leeway in self-organizing and were left wondering why no communities of practice formed. When I asked the team members about this, they told me they were under the impression that such informal groups would be frowned upon by management. Make sure your team members know that such cross-team communities are not only OK but encouraged.

Participation

Most formally recognized communities of practice benefit from designating a community coordinator. The community coordinator is not a leader of the group but does serve the community in two important ways:

• Developing the practice around which the community has formed

• Developing the community itself

The community coordinator does this by being the one to schedule meetings and other events, convincing members to attend, connecting individuals with common interests, participating in community events himself, and so on. In some respects, the role of community coordinator is similar to that of ScrumMaster. From my experience, being a community coordinator takes 5 to 20 hours per month, depending of course on the community. The community coordinator might even be a full- or nearly full-time position in a community that has been assigned formal responsibilities in the organization.

There are no rules about how many hours a community should take away from a member’s project team. It can literally vary from a few hours a year to a few hours a week. Babinet and Ramanathan describe the relatively high level of commitment on Salesforce.com’s Virtual Architecture Team.

The VAT meets for two hours twice a week to review the technical implementation of products and features being built by the Scrum teams. The teams building the most complex features in the release are asked to present to the VAT. The group provides valuable feedback to the Scrum team on how their technical design will impact or be impacted by other areas. The VAT focuses primarily on the technical implementation, especially scalability and performance considerations. Teams asked to make significant changes must present again in the same release cycle [approximately three months] and provide details on how they modified their design. (2008, 405)

Communities of practice are well worth the time and investment. The service they provide in aiding communication and coordination across a large organization or a large project is invaluable. If communities of practice have not yet formed in your organization, start one around a topic that interests you or is causing your organization pain. As that community begins to contribute to the organization, other communities are likely to form as well.

Scrum Does Scale

You have to admire the intellectual honesty of the earliest agile authors. They were all very careful to say that agile methodolgies like Scrum were for small projects. This conservatism wasn’t because agile or Scrum turned out to be unsuited for large projects but because they hadn’t used these processes on large projects and so were reluctant to advise their readers to do so. But, in the years since the Agile Manifesto and the books that came shortly before and after it, we have learned that the principles and practices of agile development can be scaled up and applied on large projects, albeit it with a considerable amount of overhead. Fortunately, if large organizations use the techniques described regarding the role of the product owner, working with a shared product backlog, being mindful of dependencies, coordinating work among teams, and cultivating communities of practice, they can successfully scale a Scrum project.

Additional Reading

Beavers, Paul A. 2007. Managing a large “agile” software engineering organization. In Proceedings of the Agile 2007 Conference, ed. Jutta Eckstein, Frank Maurer, Rachel Davies, Grigori Melnik, and Gary Pollice, 296–303. IEEE Computer Society.

This experience report describes the first couple of years of BMC Software’s adoption of Scrum on a 250-person project. It is an excellent first-person account of the struggles and rewards faced by the project’s engineering leader. It concludes with nine guidelines for success distilled from the author’s experience on the project.

Larman, Craig, and Bas Vodde. 2009. Scaling lean & agile development: Thinking and organizational tools for large-scale Scrum. Addison-Wesley Professional.

This book covers many topics, but in Chapter 11, Larman and Vodde focus specifically on large-scale Scrum. They present two frameworks for scaling Scrum: one for scaling up to ten teams, another for scaling beyond that.

Leffingwell, Dean. 2007. Scaling software agility: Best practices for large enterprises. Addison-Wesley Professional.

This book focuses on two different types of scaling: scaling agile within large organizations and scaling agile on large projects. A bit more emphasis is on the former type of scaling, but both are covered. The heart of the book, Part II, focuses on seven agile practices and how to scale them: teams, planning at two levels, iterations, small releases, concurrent testing, continuous integration, and regular reflection.

Wenger, Etienne, Richard McDermott, and William M. Snyder. 2002. Cultivating communities of practice. Harvard Business School Press.

This book is the authoritative source of information on communities of practice. Included is information on how to encourage them to form, how to lead them, how to measure the value they provide, and some of the downsides of using communities.