Photo by Arthur Rothstein. Library of Congress, Prints & Photographs Division, FSA-OWI Collection, Reproduction Number: LC-USF34-024346-D.
Elevated structure and buildings. Lower Manhattan, New York, December 1941.
To use software configuration management (SCM) properly, you need to understand how its techniques fit into the larger picture. This chapter discusses some of the other aspects of the development environment and the processes we use to create and maintain them so that we can understand how SCM fits in.
By Software Configuration Management, we mean the set of processes that one uses to create and maintain a consistent set of software components to release. Version control is the part of the SCM process that a developer sees most often. It manifests itself through your version control tool and how you use it. Often you associate the use of a tool with a set of policies or rules for how to use it to follow the lifecycle the organization needs (check-ins, builds, releases, and so on). Sometimes the policies are enforced automatically; other times developers must follow a manual procedure.
We can't give you one set of rules to say how to use version control or which tool to use. How to use it (or any other tool or technique) depends strongly on your situation. Just as you might use different tools for building a doghouse as opposed to a summer vacation cottage, different software project teams need different approaches. If you use big complicated tools for a small project where everyone communicates well, you may slow things down and increase costs. Likewise, trying blindly to apply techniques that worked well in an initial product release project in a five-person start-up may cause trouble in the same start-up six months later with 20 people working on release 3 (and maintaining release 2 and release 1).
Working in a team adds a need for communication and changes the way you would execute these principles. For a one-person project, you can try to build your software system and accompanying development environment based solely on first principles of “good” coding practice. Some basic version control system will help you with your memory, but you should not (usually) have any communication issues. Once more than one person is involved in building the system, communication and interaction between developers come into play. Each software system is built in a context that shapes the code and the patterns of work. The same principles you use as an individual developer also apply when you work as part of a team. Each team and every project are different, so no one process will work. Version control is a core part of the communication mechanism. You need to vary the details of how you use version control based on your situation. To do that, you need to step back and look at the places where software is built.
We can apply a few general principles to using configuration management on any software project. The details of how to apply the principles vary, depending on the size and nature of the team. Some of these principles are as follows.
Use version control. Version control is the backplane on which software organizations communicate work products among themselves. This sounds obvious, but some organizations do not track versions. Even if they have a version control tool, they do not use it. They copy files between developers without checking them in. They have no way of identifying what went into a release and how to reproduce the system at a point in time. Some places need more lightweight tools and processes than others, and some development techniques allow for more flexibility in the process, but every team needs to have a way to do version control and needs to use it to communicate code changes among the members.
Do periodic builds, and integrate frequently. Have some sort of periodic build process that builds what looks like the current version of the software so that people can see how well things fit together. The longer you put off integration, the harder integration problems will be. How often you need to do this depends on a number of details about your organization. Extreme Programming (Beck 2000) and other agile approaches say to integrate continuously, but for simple systems, that may cause too much overhead. Erring on the side of integrating too often is generally better, but integration takes time, and you want to find the point where integrating more often speeds up the overall process.
Allow for autonomous work. Every team, even those with one person, may need to work on various points in time of the codeline. Team members should be able to control what versions of what components they are working on. Sharing “common” components may work well most days, but when you need to diverge from working on the latest code, it will probably be for an emergency.
Use tools. If you have too many manual processes, people may make mistakes or simply skip a step out of frustration or perceived need. Be lazy and write tools.
How you apply these general rules depends on the environment you are building your software in. And you need to apply other software engineering practices to testing, coding, and related issues. See the Further Reading section for pointers. The next sections describe some elements of the software environment.
In its most concrete form, a software system is the sum of all of its code. You write code, and you end up with an application that does something useful. Other artifacts also make the system run: data, documentation, and anything else you need for your system. The code, however, defines the shape of the other artifacts. Without the code or these artifacts, there is nothing to deliver to a customer or user.
To understand how you build a software system, you need to think about more than just the code and other physical artifacts. You need to understand how the people on the team work, how the product is structured, and what the goals and structure of the organization are. We often ignore social issues such as organizational structure and politics, thinking them secondary to what we perceive to be the main goal: building a software system. Although this sounds appealing (focusing on the core goals and all that), these other factors are important to consider. If you don't consider them, they will get in the way when you try to build and deliver systems quickly.
It's easy to get lost in all the other factors. Corporate politics alone can cause you to spin endlessly. One way to think about the software environment where you are using configuration management is to model it as consisting of the following structures.
Where the developer codes: the workspace. This is the day-to-day code environment of the developers.
Where the coding takes place: the organization. The developer works in the context of an organizational structure. Teams are developing software, testing, marketing, doing customer support, and so on.
Where the code fits in: The Product Architecture. Related to the product architecture is the release structure, which specifies how many releases are being developed together.
Where the code (and code history) is kept: The Configuration Management environment. This includes tools, processes, and policies.
These structures all have associated with them policies and processes that sustain them. Although it's not always obvious, these structures influence each other (see Figure 2-1), and if you don't take the influences into account, your work may be harder than it needs to be.
Some examples of these influences follow.
The organization influences the structure of the code and the architecture (Conway's Law).
The organization and the architecture both influence version control policies. A less tightly coupled architecture can be developed in parallel more easily than a tightly coupled one. A small team in one room can communicate more effectively than a large team spread across the globe. In each case, version control policies can compensate.
The version control environment and the organization influence the structure and use of workspaces. You can use some policies to alter the way people work; this isn't always the best way to do things because it is working backward in many cases, but it can have an influence.
Often tools and mechanisms for work style and version control fail because they are applied without understanding the reality that the mechanisms need to work well with the existing environment.
The next sections discuss each of the development structures and its interactions in more detail.
Software development happens in the workspaces of the developers in the team. A workspace is simply the set of components a developer is currently working with, including source code the developer is editing and using for reference, build components necessary to run the software, and third-party components. The Private Workspace (6) pattern discusses the components of a workspace in more detail.
Each developer has one or more workspaces, depending on the number of projects the developer is working on. The build team may also have an integration workspace in which to do periodic builds.
“Software architecture” is a widely used and widely misunderstood term. It can mean many things, depending on the person using the term and the audience. Sometimes different understandings of what “architecture” means can lead to legal disputes. Steve McConnell recounts a story about being an expert witness in a case where a company sued a VP over nonperformance of work duties, in part because they had a different understanding of what an architecture was (McConnell 2000).
The one thing that most definitions have in common is that the architecture defines some aspects of the structure of the system. IEEE Std. 610.12 defines architecture as “The organizational structure of a system of components.” The architecture places constraints on where and how parts fit together and how easy it is to change or add functionality. Even if there is no explicit architecture document or architectural vision, there is an implicit “architecture” that the existing body of code defines. The code establishes a structure you need to work within, perhaps changing it as you go. Even in a “green fields” development project, there are still “architecture” decisions that constrain software decisions, including (but not limited to) method of communication, language, and choice of database.
For the purpose of discussing team software development, we define architecture as “a description of how the parts of the software fit together that provides guidance about how the system should be modified.” This includes both the logical and physical structure of the system, including relationships between components, deployment units, and even the way the code is structured in the source tree.
The role of architecture in developing a version control strategy is to establish the structure of the version control modules and therefore how we make concurrent work easier. Grinter writes that “software architectures represent another mechanism of interaction supporting collaborative work at a higher level of systems abstraction” (Grinter 1995).
The architecture influences this because the architecture defines
What makes up a deliverable unit
The communication paths among the units
Indirectly the directory structure and other structural aspects of the source code respository
For example, one way to allow concurrent work is to design a system using a pipes-and-filters architecture (Buschmann et al. 1996). At a finer level of detail, you can decouple systems using patterns such as Parser-Builder (Berczuk 1996a) or Visitor (Gamma et al. 1995). These patterns each allow teams to treat the code that they own as fairly separate from other teams' code. The pipes-and-filters approach can be a way to design a system that is being built by teams a large distance from each other. It is entirely possible to structure your source tree in a way that conflicts with this goal, and, if you ignore integration process issues, the architecture alone will not save you.
An architecture comprises a number of views, each of which exposes the part of the picture that you care about in a specific context. The UML (Booch et al. 1999) takes this approach by defining the following four views as well as a use case view that describes what the software does (Krutchen 1995).
Implementation, which defines units of work at various degrees of granularity.
Deployment, which specifies where physical components are put. The deployment structure has a strong impact on how software is built.
Design, which is the lower-level details. Except for the way the class design affects the module structure and the dependencies between components, this has the least effect on how the version control should be structured.
Process, which affects performance, scalability, and throughput. This has the least relevance to our discussion, though it is important.
Architecture is heavily influenced by organizational structure. In certain occasions an architecture can influence the formation of an organization, but most often, the product architecture is developed in an existing organization.
Architecture can be organized with the organization in mind. The architecture may match the structure of the organization, based on location and the skill levels of developers of components.
Modularity leads to decoupling, which adds concurrency to the development process.
Modularity is about separation: When we worry about a small set of related things, we locate them in the same place. This is how thousands of programmers can work on the same source code and make progress. We get in trouble when we use that small set of related things in lots of places without preparing or repairing them (Gabriel and Goldman 2000).
The architecture and organization affect the best way to partition source code into directories.
The Architecture/Module Structure. The product's module structure is the strongest influence on the source code structure. A typical partitioning is to have the source code for one module in one directory so that you can manage the files more easily. If you are using a language that has header files, such as C++, or interface definition files, such as COM, or CORBA IDL files, you may want to put these files in a common location, apart from the bulk of the source code; this makes it easier to ship interface definitions to customers if you provide an API.
The team structure—the number of people working on a project and the number of people working on a module. Modules may be grouped logically based on the people working on them. This may not always be optimal, but it happens.
The particulars of your development environment. Do you have symbolic links? What is your programming language?
This section discusses some of the aspects of an organization that affect the way software systems come into being. This is a large area, which has research papers (Allen 1997a, 1997b; Allen et al. 1998) and entire books (Weinberg 1991; Brooks 1975) devoted to it, so we mention only some of the more relevant factors here.
Software development is, in many respects, a social discipline. The organization can significantly influence how a product is designed and how teams are structured. Things can get difficult when your development approach conflicts with the structures the organization imposes. Changing the organizational structure is often the best, if most difficult, fix. Even if changing the organization is impossible, the real danger is in not acknowledging these influences and in focusing only on the immediate programming tasks.
Organization has an influence from the perspective of workspace structure and version control because the structure of the organization constrains communication. The organization affects communication by the distance between people and teams. Distance is a measure of how hard it is for teams and team members to interact. It can be about physical distance but is not always. Organizational structure divides responsibilities, which can also make communication harder if the responsibilities are not divided in a manner consistent with the needs of the application.
The organization can be more subtle in its influence on the architecture and the way you work. The nature of the organization and its culture constrain the team dynamic, the architecture, the goals of the development process, and how problems are handled.
Distance is about the following:
Physical location. Teams that are not physically close can have a lower bandwidth of communication.
Culture and team dynamics. An appropriate culture can result in teams that are physically far apart having very good communication and can also result in people in the same room having very poor communication.
Organizational structures that dictate communication paths. This is an aspect of culture since communication need not follow corporate structures, but if this is an ethic in your organization, you might do well to have the architectural communication paths follow the organizational boundaries.
You need to be aware of the effects of organizational distance, and we encourage you to be a change agent to improve the long-term picture, but simply building your work structures so that they are robust in the face of these organizational issues will generate a big win.
Some other organizational influences include
Skill sets of people
Distance, which encompasses many things, including physical distance
Values and culture
Location of personnel and other resources
Team stability
Type of company: consulting company versus product company versus product company with customer-specific changes
The structure of an organization can have a large impact on how the software is built and hence on coordination needs. Organizational forces can have a very strong impact on the product architecture and can control process aspects such as when and how releases are created, tested, and so on. It is beyond the scope of this book to tell you how to match the product process and architecture to adapt and leverage the structure of the organization. We briefly describe some of the issues so that you can better understand them.
Although many say that an ideal development environment has developers who are near each other and who communicate effectively, many real organizations have resources that are geographically distributed, and you need to help them work well together. One way is to distribute responsibilities so that remote groups can have very well-defined interfaces between them and few dependencies. You also need to structure the version control system so that people at all locations can see all the code easily.
Some influences of organization on architecture that are particularly relevant are the following.
Module structure (Bass et al. 1998). Modules should be developed by people who can work well together, and aspects such as geography or technology experience may dictate that certain components should be developed by certain groups. There is an interplay with the product architecture here; you may have to make the choice of assigning a component to a group with the most appropriate technical experience, for example, or a group with lesser technical skills but in a better position to interact (by whatever means are appropriate—this is not an argument for geographic proximity) with the groups that interface with its work product.
Integration Policy: How often all of a system's components are integrated is often a function of organizational policy.
Workspace management: How you set up your local development environment and how your workspace relates to others'.
Version control and identification: How you use source control tools and other means to coordinate changes with others, publish your changes, and reproduce environments, such as when you need to fix a bug in an earlier release. This includes issues such as branching and labeling, which often cause much consternation.
Coordination: How you work together with other teams and developers.
Identification and Auditing: How you know what you built.
Given all the things that have a part in building a software system, it is easy to lose track of your primary goal, which is to build a software system. It is easy to become overwhelmed by the technologies and techniques you use as part of the process. Version control, configuration management, building, testing, pair programming, branching, and other process activities are all aspects of the software engineering environment. These techniques support the process. A goal is not to branch, for example. Branching is one way to accomplish concurrent development or isolate a line of work. This may seem like an obvious statement, but I have seen a lot of energy expended on figuring out how to accomplish tasks for which there were other, simpler ways to reach the end goal. An excellent bit of advice to keep in mind as you read this book is, “Don't mistake a solution method for a problem definition, especially if it is your own solution method” (Gause and Weinberg 1990).
Putting together all the influences of architecture, organization, and configuration management on each other, we see that the big-picture view of an SCM environment encompasses the tools and process for identifying, organizing, controlling, and tracking both the decomposition and recomposition of a software system's structure, functionality, evolution, and teamwork. An effective SCM environment is the glue between software artifacts, features, changes, and team members.
Steve McConnell has a few books that cover best practices for coding and teamwork—in particular, Rapid Development (McConnell 1996) and Code Complete (McConnell 1993) have some excellent guidelines.
The Pragmatic Programmer (Hunt and Thomas 2000) talks about the value of using tools to automate processes and procedures.
The Practice of Programming (Kernighan and Pike 1999) says quite a bit about the value of automation and tools.
Mary Lynn Manns and Linda Rising provide some guidance on introducing new ideas in their paper, “Introducing Patterns into Organizations” (Manns and Rising 2002).
Alistair Cockburn gives some good advice about writing use cases in Writing Effective Use Cases (Cockburn 2000). Patterns for Effective Use Cases (Adolph et al. 2003) provides a patterns-based approach to writing use cases.
David Dikel and David Kane's book, Software Architecture: Organizational Principles and Patterns (Dikel et al. 2001), is an excellent source for information about what architecture is and how to use it.