Library of Congress, Prints & Photographs Division, Detroit Publishing Company Collection. Reproduction Number: LC-D418-31625 DLC.
Construction, Grand Central Terminal. New York, between 1905 and 1915.
You have an evolving codeline that has code intended to work with a future product release. You are doing most of your work on a Mainline (4). When you are working in a dynamic development environment, many people are changing the code. Team members are working toward making the system better, but any change can break the system, and changes can conflict. This pattern helps you balance stability and progress in an active development effort.
You develop software in teams because you want there to be concurrent work. The more people you have working on a code base, the more you need communication among team members. You also have more potential for conflicting changes.
For the team as a whole to make progress, you need synchronization points where the work comes together. As in any concurrent system, having a synchronization point means that we might have deadlock or blocking if we don't manage the coordination correctly. Deadlock can happen when two people have mutual dependencies, and a check-in before a test finishes could mean that the test would fail the next time we ran it. Blocking can happen when the precheck-in process takes too long and someone else needs the changes to proceed.
If we think of software development as a set of concurrent processes, we have to synchronize whenever we check in a change to the codeline in the source control system. When someone checks in a change to the codeline, that person can cause delays for the whole team if the change breaks someone else's work. Even if we tested the change with what we thought was the latest code, it could still be incompatible with the change that was checked in moments before. But if we put too much effort into testing for changes, making sure that a change works with every change that was checked in while we were running the last set of tests, we may not be able to check in our changes in a reasonable amount of time. That can cause delays too.
Working from a highly tested stable line isn't always an option when you are developing new features. You want to use your version control system to exchange work in progress for integration, and there may be many features that you don't have integration tests for yet because they are so new.
You want to be able to get current code from the source control system and have a reasonable expectation that it will work. You want the codeline to be stable so that it does not interfere with people's work. You can require that people perform simple procedures before submitting code to the codeline, such as a preliminary build and some level of testing. These tests take time, though, and the delay between check-ins may work against some of the project's greater goals. A broken codeline slows down everyone who works from it, but the time it takes to test exhaustively slows down people as well, and in some cases can provide a false sense of security because you can never fully test a system that is in flux. Even if you do test your code before check-in, concurrency issues mean that two changes, tested individually, will result in the second one breaking the system. And the more exhaustive—and longer running—your tests are, the more likely it is that a noncompatible change may be submitted, as Figure 5-1 shows.
You can prevent changes from being checked in to the codeline while you are testing by using semaphores, but then only one person at a time can test and check in changes, which can also slow progress. Figure 5-2 shows a very stable but very slowly evolving codeline.
You can also make changes to your codeline structure to keep parts of the code tree stable, creating branches at various points, but that adds complexity and requires a merge.
You can go to the other extreme and make your codeline a free-for-all. Figure 5-3 shows a quickly evolving but unusable codeline.
You can also change the module architecture of the system to reduce the likelihood of conflicting change, but even then you may still have two people changing the code in a way that breaks something.
Aiming for perfection is likely to fail in all but the most static environments. You can achieve stability on a given codeline but with process and synchronization overhead, increased merging, and more complicated maintenance and administration. This is not always worthwhile. You want a balance: an active codeline that will, more likely than not, be usable most of the time.
Note
Institute policies that are effective in making your main development line stable enough for the work it needs to do. Do not aim for a perfect active development line but for a mainline that is usable and active enough for your needs.
An active development line will have frequent changes, some well-tested checkpoints that are guaranteed to be “good,” and other points in the codeline that are likely to be good enough for someone to do development on the tip of the line. Figure 5-4 shows what this looks like.
The hard part of this solution is figuring out how “good” your codeline needs to be. You need to go through a process similar to doing a requirements analysis for building a software system. Your clients want perfection and completeness, and they want it quickly and cheaply. These goals are, in reality, unattainable. Do an analysis along the following lines.
Who uses the codeline?
What is the release cycle?
What test mechanisms do we have in place?
How much is the system evolving?
What are the real costs for a cycle where things are broken?
For example, if the codeline is being used by other teams that are also doing active development, some instability is appropriate, and the emphasis should be on speed. If this is the beginning of the new development or if the team is adding many new features, you expect more instability. If this codeline is basically stable and being used as a standard component, more validation is appropriate. Right before you want to branch or freeze for a release, you want more stability, so you want to test more.
Understand your project's rhythm. Dikel and Kane define rhythm as “the recurring, predictable exchange of work products within an architecture group and across customers and suppliers”(Dikel et al. 2001). A good project rhythm is especially important for architecture-centric development, but any project that has concurrent work with dependencies needs a good rhythm. The source control structure can influence how the rhythm is executed, and culture helps define what rhythm you need.
If you have good unit and regression tests, run either by developers or as part of the system build postcheck-in, errors will not persist as long, so emphasize speed on check-in. If you do not have a good testing infrastructure, be more careful until you develop it. If you want to add functionality, emphasize speed.
If clients need a good deal of stability, they should use only Named Stable Bases (20) of the components, enabling them to avoid “cutting edge” work in progress. Figure 5-5 shows how these baselines are identified and labeled when they pass tests. But these clients should then be treated more like external clients than members of the active development team.
Don't be too conservative. People can work with any system as long as they understand the trade-offs and the needs. You don't want to make the check-in process too difficult. If you have a precheck-in process that takes a long time, you run the risk of developers doing larger-grained and less frequent check-ins, which will slow feature progress. Less frequent check-ins increase the possibility of a conflict during testing and make it harder to back out a problematic change.
Establish criteria for how much to test the code before check-ins.
The standard needs to set a quality level that is strict enough to keep showstopper defects out of the daily build, but lenient enough to disregard trivial defects (because undue attention to trivial defects can paralyze progress) (McConnell 1996).
Martin Fowler suggests for the purposes of continuous integration (Fowler and Foemmel 2002) that a selected suite of tests run against the system successfully. The more exhaustive the tests, the longer the precheck-in time. You need to determine how much stability is really necessary for your purposes.
Remember that there is a fundamental difference between code that is close to release and code that is being actively changed. If you need a stable codeline, perhaps what you want isn't the active development line but a fully tested and verified Release Line (17). There are significant benefits in the form of catching potential problems early in developing with an ActiveDevelopment Line (5). You can also push off your more exhaustive testing to a batch process that creates your Named Stable Bases (20).
To prevent total chaos on the mainline, set up each developer with a Private Workspace (6) where each can do a Private System Build (8), Unit Test (14), and Smoke Test (13).
Have an integration workspace where snapshots of the code are built periodically and subjected to more exhaustive tests.
Any SCM tool that supports “triggers,” or automatic events that happen after a change is submitted, will help automate the process of verifying that you are meeting the quality metric. You can then set up the system to run a build or a set of tests after a change is submitted. You can also set up the system to run less often.
As Jim Highsmith writes, “Change is expensive, no question about it. However, consider the alternative—stagnation” (Highsmith 2002).
Once you have established that a “good enough” codeline is desirable, you need to identify the codeline that will be like this. Codeline Policy (12) will establish which lines follow this form and what the check-in/commit process is for these (and other) codelines.
Individual developers still need isolation to keep the Active Development Line (5) alive. They can do this by working in a Private Workspace (6).
When the need for stability approaches, some work will need to be broken off to a Release-Prep Code Line (18).
Some long-lived tasks may need more stability than an active development line can provide, even though you realize that there may be an integration cost later. For these, use a Task Branch (19). Doing this also insulates the primary codeline from high-risk changes.
One reason people resist applying this pattern is that they think their problem is that code is not perfect, when in fact the problem is that it is too hard to change and evolve the code. A great book about getting to the core of the “real” problem is Are Your Lights On? (Gause and Weinberg 1990).
Agile Software Development Ecosystems (Highsmith 2002) discusses the reality of continuous change in most projects.