Chapter 11. Discovering Patterns

How do we discover and identify patterns? We can gather patterns from external sources such as vendors and the community easily enough. However, things get more complex when we seek to discover and identify the patterns that are unique to our organization. We also need to keep in mind that these two aspects—internal and external sourcing—do not happen in isolation. We strive to create a set of patterns that draws upon both sources and delivers a mix that is unique and tailored to the needs of our organization. The goal is to define a set that delivers a competitive advantage to the organization.

To succeed in discovering the patterns that we need, we can use the patterns and guidelines in this chapter to help us answer questions such as these:

• Patterns come from multiple sources; how do we find them?

• What are the best practices associated with finding patterns?

• Who should be involved? What should they do? When should they do it?

• Although numerous pattern opportunities can and will be identified, not all will be developed. How do we determine which patterns need to progress beyond just possibilities? Where do we make the investment? How do we decide?

• In the search for patterns, we don’t always start from scratch. How can we leverage existing patterns as we try to identify and discover new ones?

Patterns

In this section we look at the patterns that help us to figure out which patterns should be created. We examine how to identify opportunities for using and creating patterns. We also need to consider solutions that may serve as the basis for patterns, ideally best practices but possibly also worst practices.

Antipatterns

Context

The organization struggles to ensure that the entire team is following best practices consistently. In addition, testing and code reviews reveal that common mistakes are occurring.

Problem

How do we capture and then share common worst practices with the team? How do we ensure that the team is aware of and then avoids repeating these worst practices?

Forces

• A cultural change is needed that values the identification of worst practices. Time is spent on identifying, detailing, and then communicating worst practices. This may seem counterintuitive, since we expect investments to be made only in positive practices.

• Investment in antipatterns is similar to that made in patterns. A successful effort requires training, tooling, and time.

• Efforts related to antipatterns are usually more effective when linked with patterns that would help mitigate the antipatterns.

Solution

As a parallel effort to using patterns, we also need to identify, detail, and communicate antipatterns. An antipattern is a proven worst-practice solution applied within a given context.

There are a number of different approaches to identifying antipatterns: leveraging experience, harvesting from existing solutions, code reviews, and so on. As we identify antipatterns, we need to look at the ROI for creating the specification associated with the antipattern. For those antipatterns that justify the investment, we should create a specification and then make it available to the team via the asset repository, just as we do with pattern specifications.

In addition, as we document our development process, we should be certain to update the process to highlight cases where certain antipatterns may occur or are prevalent. We should also identify and document the patterns and guidelines that help mitigate or correct these antipatterns. Identifying an antipattern may be a good indication that there is a need for a pattern.

We can develop antipattern implementations that allow us to automatically detect the use of antipatterns. These can be useful for supporting automated reviews as part of a continuous integration effort.

Example

Within PBE there are a number of antipatterns:

• Perfect Pattern. Pattern producers spend a great deal of time accounting for all possibilities and points of variability as they create a pattern. As a result, the time frame to deliver the pattern is much longer than necessary (and sometimes the pattern is never released). This is similar to the “analysis paralysis” antipattern that can happen in software development projects. To counteract this antipattern, consider using the Piecemeal Pattern Creation pattern (Chapter 10) and the Update Existing Patterns guideline.

• Generic DSL. Software engineers spend a great deal of time and effort in making a DSL as generic as possible. Their logic is that doing so insulates them from changes in the underlying layers of the solution and is an investment in the future. However, this greatly diminishes the value of the DSL as it no longer is specific to the domain. If a general-purpose language is needed, it is best to use a generic language. To counteract this antipattern, consider using the Integrated Patterns and DSLs pattern and the Design a DSL guideline (Chapter 15).

• Top-Down DSL Creation. In designing a DSL that is directly tied to the domain, little consideration is given to how the language will be used. When we use models for design, communication, and generation, we need to be aware of how they will be consumed and leveraged. To that end, as we design the DSL, we need to ensure that we consider bottom-up constraints. To counteract this antipattern, consider using the Meet-in-the-Middle Pattern Creation pattern (Chapter 12) and the Design a DSL guideline (Chapter 15).

Related Patterns and Guidelines

Determine Business Impact*, Pattern Opportunity*, Single Pattern–Varied Use Cases (Chapter 10)

Pattern Opportunity

Context

A software project is under way for which requirements (use cases, user stories, etc.) have already been identified. The team is working through the current iteration.

Problem

Although we want to use patterns on a project, we are not sure what patterns to use. Perhaps the patterns we are aware of are either too generic or not the right fit for the current project. Overall, there is a lack of awareness regarding the solution being built and opportunities for creating and using new patterns. As a result, it looks as if similar and repetitive work will be performed manually, and likely not using best practices. How do we identify the opportunities for using patterns on our project?

Forces

• We need to spend time evaluating which patterns are worth pursuing, as we will likely identify more patterns than it makes sense to invest in.

• We will have to spend time analyzing the solution we are building.

• A cultural change accompanies PBE, whereby the team becomes conditioned to look for pattern opportunities.

Solution

As we start a project or iteration, we include time to consider how patterns could be used to help in creating the solution. Opportunity identification is all about looking for and finding opportunities where we can use patterns. We need to tune our creativity and sleuthing skills to hunt for pattern opportunities. These opportunities may involve patterns from external sources, those already created internally, or patterns that need to be built.

We need to look for situations where the same solution is used multiple times. We need to be able to use abstraction and simplification to see the true solution in situations, rather than just the details associated with each separate situation. We need to be able to uncover patterns that provide the best solution for the problem and context.

To start, we should be armed with the right questions:

• Have we already solved this problem?

• Will this solution need to be reused elsewhere? Which iteration of the project? Which project?

• Has someone else on the team already solved this problem?

• Does my work on this use case relate to the work that others are doing on their use cases?

• Are there common aspects of a solution that would apply across a number of use cases?

• Are there best practices associated with the frameworks that we’re working with?

• In bringing together a number of frameworks, will any best practices emerge?

• Are there rules, assumptions, and constraints that would limit the amount of information that actually needs to be provided here?

• Do we have guidance on how to solve problems captured only as documentation? Do people follow the documentation?

This pattern provides a key mechanism for leveraging and engaging the team, providing them with an opportunity to be creative problem solvers. It will challenge them and require them to be creative, insightful, and innovative, always on the lookout for a better way of getting things done. Adopting such a culture makes for a much more enjoyable working environment. As we adopt and roll out PBE within our organization, it is important to make this clear to the team and to emphasize the importance of their role in making the program a success. In addition, as we work with them to find patterns, we also ensure that those who use the patterns will apply the solution consistently and rapidly. Ideally, everyone on the team contributes to this effort. Anyone can spot an opportunity, even if he or she doesn’t know how to create or detail the pattern.

Note that this pattern is focused on finding the opportunity where a pattern may be applicable. We’re not yet worried about how to write up the pattern as a specification or codifying the pattern in an implementation. Really, we are looking for candidate patterns. We’ve done some initial vetting of a pattern, but a full analysis of its impact and how to create it is not yet done at this point. We capture each opportunity that is identified in a candidate pattern list. As the project progresses, we will evaluate the candidate patterns and determine which warrant investment and should be developed.

This is a lightweight pattern that can easily be incorporated into our existing development process. The goal here is not to seek out all answers or perfect patterns. As the name of the pattern suggests, we are looking for opportunities. This pattern needs to become ingrained in the culture of the team and organization.

Common Pitfalls

A number of common pitfalls arise when following this pattern:

• Abstraction blinders. When creating a solution, we fail to recognize situations where we should be using patterns. We’re so focused on what’s directly in front of us that we fail to see what else is possible.

• Waterfall pattern use. We take some time early in the project to identify a set of existing patterns to use through the duration of the project. All of the pattern identification work is done at the beginning of the project. This inappropriately limits the patterns that we can use on the project.

• Get ’em next time. In some cases we wait until the end of the project to review and then try to harvest a set of patterns based on recurring solutions that occurred during development. This approach eliminates the benefits that can be achieved from the patterns on the current project. In addition, the harvesting approach is likely to fail in getting the assets built, because after the project the team members will be moving to new projects and tasks.

Example

In Chapter 4 we follow Oslec’s efforts to identify pattern opportunities. The identification of the candidate patterns is a team effort, and many people and roles are involved. Although they have some specific meetings early in their efforts that focus on pattern opportunity identification, the team remains on alert for pattern opportunities throughout the project.

A number of candidate patterns are identified and captured in a candidate pattern list. The team reviews the list, evaluates the candidate patterns, and determines that the Subsystem Façade pattern is most important and should be developed.

Related Patterns and Guidelines

Antipatterns*, Determine Business Impact*, Recurring Solution*

Recurring Solution

Context

We have decided that we would like to find, build, and use patterns on our projects.

Problem

Although we want to use patterns that are specific to our situation and are looking for them, we are not sure when and where the best opportunities to create patterns exist. Where do we find patterns?

Forces

• The team must communicate and be looking at how to produce higher-order solutions.

• Expertise, creativity, and problem-solving skills are more valuable than ever.

• Traditional efforts such as documentation and code reviews increase in importance.

Solution

Looking for recurring implementations of solutions is an important mechanism for creating patterns. We can use a number of approaches:

• Rule of Three. In applying the Rule of Three, we are looking for situations where the same problem/solution set has occurred in three unique instances or situations. The logic behind the Rule of Three is that “. . . the first occurrence shows the design can work, the second occurrence is interesting, and the third occurrence suggests that the design might be worthy of being a pattern because it appears to have a wider applicability.”²

• Déjà vu. Less formal than the Rule of Three, a sense of déjà vu may alert us to a possible pattern. Do we sense that we’ve seen this solution before? Does it seem that the work we are doing is similar to something that we’ve done in the past? Does it sound like something that someone else has been working on? Whether we call it an instinct, déjà vu, or a sixth sense, we need to be aware of when we sense that a solution has already appeared for the current problem.

• Experience. As we work on more projects and encounter more situations and solutions, we carry with us our own inventory of proven solutions. Over time, we will start to see some of those solutions reappearing. A brief journal, log, or other similar mechanism is useful for keeping track of solutions that we think have the potential to reappear. Again, we want to keep things as lightweight and simple as possible.

• Abstraction and simplification. Being able to see beyond the details of the current problem and its solution and recognize higher-order elements at play is a key capability. In addition, we need to be able to simplify the solutions that we plan to build.

In reality, we will likely use a number of these approaches as we start to train ourselves to look for patterns. There is no single approach to finding patterns; however, having an open mind and the right attitude and culture can take us a long way.

Regardless of the approach that we take, a key aspect to keep in mind is that communication among the team members is more important than ever. Whether we have daily Scrums, frequent design reviews, study groups, or presentations on designs, we will want to encourage and support the team’s discussion and sharing of ideas about work that they have under way. Without such communication, how will the team start to spot recurring solutions beyond the scope of their immediate deliverables?

Example

We see an example of using this pattern in the case study when the team elaborates the Subsystem Façade pattern based on the realization that in the past they have often used the Session Façade pattern along with the Data Access Object pattern.

Related Patterns and Guidelines

Antipatterns*, Pattern Creation Lifecycle (Chapter 12), Pattern Opportunity*

Guidelines

In the following guidelines we focus on advice to help determine which candidate patterns warrant an investment. We also discuss how we should capture sufficient details about the candidate pattern to support further investigation and development while not burdening the effort with too much formality. And last but not least, not all pattern investments need to be made in new patterns. We also need to consider how and when we should evolve and update existing patterns.

Determine Business Impact

Summary

Creating patterns for reuse requires an investment; how do we pick which patterns justify the expense? How do we decide which patterns transition from being candidate patterns to patterns that we will invest in and develop? Organizations have limited resources, so we can never build everything; we need to pick the right patterns to build. We need criteria and mechanisms that can help us decide and direct our investments.

Introduction

In following patterns such as Pattern Opportunity, we will find that the effort leads to the identification of a number of candidate patterns. Limited time and resources mean that not every candidate pattern can be developed. Here are some considerations to keep in mind for deciding which patterns warrant an investment:

• Much as for other development tools, we will come to see patterns as an investment that needs to be justified.

• Pattern identification and valuation efforts will start to show up on project plans.

• Regardless of whether a pattern is classified as related to the business domain or the technical domain, it will need to have a positive impact on the business.

• Pattern quality becomes valued higher than pattern quantity.

Explanation

We cannot build all of the patterns that we identify, so we need to use criteria and mechanisms to direct our investments.

A first step to consider is that when building a pattern, we need to tie it to key business efforts and results; our effort should be business-driven. Even patterns that have a technical impact need to be traceable to the impact that they will have on the project and the business. We need to be able to draw a connection from that pattern to the positive impact it will have on the business. Note that improving the time to market and increasing productivity are legitimate business goals. If we cannot determine that the pattern positively impacts the project, it is not worth pursuing any further. As discussed in the context of the Pattern Opportunity pattern, we want to be innovative. However, that innovation needs to matter and add value to the business.

So if we can trace the pattern to the areas where it will impact the business, we can move forward and start to further detail the value and ROI for the pattern. A key input into this step is to figure the pattern’s cost; considerations include the following:

• How long will it take to build the pattern? In addition to looking at building the pattern specification or implementation, we also need to look at any efforts associated with truly defining the best-practice solution that will serve as the exemplar for the pattern. We may have to spend time completing the solution, refactoring the existing solution, or expanding the scope of the existing solution.

• Does the team have the skills needed to create the pattern? If not, how much training will they need? Will they need new and additional tooling?

• Will people need to be trained in using the pattern? If so, how many? Could the pattern be made more consumable? If so, what is the trade-off in costs between training and improving consumability?

• Will the team’s development process need to be updated because of the pattern? What is the expected cost of that effort?

• Are any new tools needed to leverage the pattern? Could this expense be avoided if the pattern is not created?

• What costs are needed to deploy, manage, and support the pattern? The costs associated with the pattern go beyond just the initial development. Although these costs are often relatively fixed, we still need to account for them.

• What does past experience tell us about building patterns of similar size and complexity?

We also need to consider the other side of the ledger. Considerations related to determining the expected benefits of using the pattern include these:

• How many times will the pattern be used on the current project?

• Is it likely that the pattern will be used on future projects? If so, how many times could it be used in the future? Do we need to discount this benefit?

• What requirements will be addressed (even partially) by the use of the pattern? How important are these requirements? What is the likelihood that these requirements will change?

• Can the investment in the pattern be recouped on the current project?

As we look at the ROI, we also need to keep in mind the funding structure for the organization that is building the patterns. Will the pattern producers be part of the overall project team? Or will they be a separate team that supports all of the other development teams? Will they be seen as a profit center or a cost center? How will teams fund the development, maintenance, and support of the patterns once they are built?

An example of using this guideline is provided in Chapter 4, where the Oslec team reviews their candidate pattern list and evaluates the Subsystem Façade pattern. They give consideration to the impact the patterns identified will have on their project as well as the cost that will be associated with developing the pattern(s).

Related Patterns and Guidelines

Pattern Opportunity*, Update Existing Patterns*

Pattern Description

Summary

How do we capture a sufficient level of detail about a candidate pattern to support evaluation and feed into design activities without overinvesting in these efforts? Create a pattern description to capture initial details about a pattern that can be used to evaluate the pattern and then serve as a starting point for designing the pattern.

Introduction

In identifying candidate patterns we have started to recognize and find details about the potential pattern. We also know that as the patterns move along the workflow from candidate patterns and then into design and development, we will have to capture details about them. However, a balancing act is required. We need to capture these details while not overinvesting or making the effort overly formal.

How can we collect enough information about a pattern to evaluate it and support design activities without spending too much time on these efforts? As we attempt to answer this question, there are a number of considerations:

• We need a standard approach to capturing details for candidate patterns.

• Defining an approach to pattern descriptions increases the formality of how we identify and evaluate candidate patterns. Thus, we are increasing the investment in the PBE effort.

Explanation

The pattern description has a few fields that are similar to those found in a pattern specification. However, the pattern description is a much less formal, precise, and complete artifact. The description is used only as a starting point for creating complete and formal representations of the pattern.

A typical pattern description includes the following information:

• Pattern name. A name for the pattern.

• Summary of the context. In what contexts should we consider using the pattern?

• Summary of the problem. What problem is the pattern expected to solve?

• Summary of the solution. How does the pattern solve the problem?

• Name(s) of the SME(s). Who are the experts who can explain the pattern and assist in its creation?

• List of related patterns. What other patterns are expected to work with this pattern?

• Areas of applicability. Where in the current project do we expect the pattern to be used? What about in other projects?

This artifact is used early in the process. We want to balance the effort so that we are providing sufficient detail to help with the evaluation process while not over-investing. For those candidate patterns that get approved for further development, we will have a good place to start. We also need to remember that the content listed in the description will change and evolve.

Starting in Chapter 4, the Oslec team follows this guideline to create a pattern description for the Subsystem Façade pattern. The pattern description is completed in Chapter 5, and then it’s used as an input when they design the Subsystem Façade pattern.

Related Patterns and Guidelines

Determine Business Impact*, Pattern Creation Lifecycle (Chapter 12), Pattern Opportunity*

Pattern Harvest

Summary

How do we capture and elicit best practices, in a more consumable form, from completed and proven projects? Use harvesting to capture the details of the best practice and put them into a pattern.

Introduction

The organization has already completed numerous projects, and we are certain that there are best practices within the code. We want to use these proven best practices in future projects. How do we capture them, in a more consumable form, from those projects? Considerations to keep in mind include these:

• Capturing best practices requires seeing beyond the specifics of a particular solution.

• We must view an existing code base as both an asset for the deployed system it represents and an asset for future patterns and systems.

Explanation

Harvest the details of the best practice and create a pattern with them. With harvesting, we consult existing, completed artifacts as sources for building out our library of patterns. As we look at the existing solution, we need to keep the following in mind:

• Is this truly the best-practice solution?

• In what situations would this solution be applicable? Does this situation occur often?

• What are the points of variability that need to be added to the solution as it moves from an instantiation of a solution to a pattern?

• What is the scope of the best-practice solution? What/where are the boundaries?

• Is this truly a single best-practice solution? Or have multiple best-practice solutions been used together to create a larger solution? If there are multiple best practices, do they have value individually? Would it make sense to use them individually?

• How common is it for this solution to be used?

We can use a number of approaches to figure out what needs to be put into the harvested pattern (specifications or implementations):

• Documentation. Consult the documentation for the solution. This should provide additional insights into the context in which it was used and why it was built the way it was. This can help guide others in successfully using a pattern based on such a solution in the future.

• Automation. Use known pattern definitions to feed into automation to find occurrences of those patterns within previously built solutions. Note that this may also lead to identification of compound patterns as well as adjacent patterns. It can also lead to a deeper understanding of the solution and support us as we create new patterns.

• Visualization. Looking at the code for past solutions can be quite time-consuming. In addition, the models and design specifications for a solution often are not updated as the project evolves. In such cases the artifacts are not the true source of information for a project once it concludes. Using visualization, where we create simple visual representations with UML, allows us to get a higher-level perspective on the elements in the project and how they relate. Visualization can often be done in an automated fashion, as there is a very direct mapping between the underlying element and its visual representation. Visualization helps to hide some of the details and lets us focus on more important aspects of the solution. This can improve the productivity and quality of the work that is done in harvesting patterns. Ideally, the visualization mechanism understands definitions of existing patterns.

• Code reviews and inspections. Part of the time spent in code review and inspections should be used to find, discuss, and then document the best practices that are present in the code. Leverage this effort and ensure that this knowledge is captured for use in the possible creation of a pattern from the solution. We are particularly interested in the rationale that goes along with the solution; for instance: Why and when should it be used? What are the implications of using it? Are there other related solutions (either complementary or mutually exclusive)?

Note that there is a momentum aspect to using Harvest Patterns. As we become more adept at identifying, documenting, and harvesting patterns, we will pick up speed. For instance, as we define a larger set of patterns, it will become quicker, easier, and more productive to run those definitions through pattern detection engines.

Related Patterns and Guidelines

Determine Business Impact*, Exemplar Analysis (Chapter 12), Recurring Solution*, Use Patterns to Find Patterns (Chapter 16)

Update Existing Patterns

Summary

Best practices evolve over time as technology and approaches mature and new innovations are introduced. How do we ensure that the collection of patterns within our organization continues to represent best practices? We need to keep in mind that creating new patterns is only one approach. Another approach is to update existing patterns.

Introduction

We have a collection of patterns that we use within the organization. We want to keep them up to date and valid. Best practices evolve over time because of changes in technology and new innovations. How do we ensure that our patterns continue to represent best practices? Some considerations include the following:

• We need to continue to evaluate which patterns warrant investment and support over time.

• Not all situations call for the development of a new pattern.

Explanation

As we look to enhance the collection of patterns we have available within the organization, it is important to keep in mind that creating new patterns is only one approach. Another approach is to update existing patterns as necessary to adapt to new technology or to evolving practices. In this way we leverage the investment that has already been made in the pattern and extend the time frame over which this investment pays off. Recognizing and planning for updates to existing patterns also helps us to avoid antipatterns such as Perfect Pattern.

Update Existing Patterns is related to the PBE Core Value of make your patterns live. A pattern is not created and then left alone, never revisited, fixed, or enhanced. After we make the pattern available, we need to continue to monitor the use of the pattern, best practices, and the associated technologies related to the pattern. Over time, one or more of these aspects (and likely others) will change and have an impact on the value and importance of the pattern. As the aspects and conditions surrounding the pattern change, we need to monitor the situation and update the pattern as appropriate.

The main focus of the effort is ensuring that the pattern is up to date, supporting the current best practices of the organization. Why and when should we update? Some scenarios include

• To address defects/bugs

• To address enhancement requests

• Changes to best practices

• Changes to associated technology

• To improve the design of the pattern

• To increase the scope of the pattern

As always, we need to ensure that the investment made in the update effort is justified, based on the value provided by the pattern.

Related Patterns and Guidelines

Determine Business Impact*, Piecemeal Pattern Creation (Chapter 10)

Summary

In this chapter we’ve focused on the efforts associated with determining the patterns we should use and build within our projects. This has included a look at how we figure out what patterns we can use, as well as how we can analyze the list of candidate patterns to figure out which patterns truly add value and contribute to the success of the business.

The patterns and guidelines in this chapter play a critical role in making PBE a systematic and disciplined approach to using patterns. Taking an ad hoc approach to finding, building, and using patterns is a sure way for a patterns initiative to fail. Without a clear analysis of what patterns are needed and the supporting ROI justifications, how can we support the claims that an investment is needed?

Most organizations already have accumulated numerous artifacts that can serve as the basis of patterns. We can leverage the expertise that exists in our personnel, existing solutions, and documentation to create patterns. In addition, we can take the opposite approach: What are the current worst practices? We need to be aware of these antipatterns as well, making sure that the team avoids them and their associated results.

Ultimately, business impact is critical. As we evaluate patterns and antipatterns, we need to focus on how these assets help us deliver the solutions needed by the business, and how they can help us do so better than other available alternatives.