Integration problems have been with us since the advent of information systems. What has changed is the nature of integration. In the past, it was focused on making information systems work together at a hardware or software level. The consumer of the integration was the technology professional. Today integration is required to directly support an organization's goals and needs.
This book is intended to help organizations deal with this change and align an organization's strategy and tactics with the ability to effectively implement a solution. Given the importance of integration, it is imperative that the IT organization is able to quickly and efficiently implement a solution. The role of an integration architecture is to provide reusable infrastructure services to support the rapid implementation and reduce the cost of new business solutions. The integration infrastructure ultimately enables business agility.
Organizations that recognize the importance of an agile infrastructure and are willing to make the investment will choose to apply the strategic approach. Organizations with more immediate needs and/or tight budgets often follow the tactical approach. In either case, as we have seen with best in class organizations, it will be imperative that an IT organization becomes effective at leveraging their existing assets.
In the end, this book is about return on assets. The critical concepts to getting the most out of your assets are:
The accelerating rate of business change, combined with the highly competitive global marketplace, has forced organizations to find new ways to respond quickly and effectively. Service-oriented architecture (SOA), considered best practice for decades, is finally being widely adopted. SOA is becoming an architectural imperative for enabling business agility and maximizing ROI of IT assets.
Reducing redundancy in the infrastructure significantly reduces the total cost of ownership (TCO). A significant amount of energy goes into duplication of effort around the technology used to integrate. As a result, each integration takes longer than necessary, does not build significant competency, and leaves the organization with an operational complexity that makes future integration even more difficult. Ultimately, this impacts the bottom line in higher maintenance costs.
Many business problems are never addressed because the integration complexity cuts across the entire enterprise and there is no effective mechanism to integrate other than to “boil the ocean.” Infrastructure should be built and used to be effective. In many organizations infrastructure is bought but never fully implemented or adopted by the organization. It is critical that the organization embrace the infrastructure to get real business benefit.
Proprietary technology has been one of the biggest limiting factors that have held back effective enterprise integration. The emergence of standards is paving the way to developing a truly open integration infrastructure that supports all platforms, vendors, and programming languages. XML and Web services standards will do for integration what TCP/IP did for networking. At this point, there is more risk in not adopting and implementing these standards.
Integration is inherently complex. It involves numerous technologies and organizational entities. Companies need to manage both the technical and organizational complexity in order to ensure success. This includes reducing redundancy in the infrastructure, training personnel in the effective use of the technology, and providing a structure and process for resolving cross-organizational issues and turf wars that may arise during the integration process.
Although integration technology plays a big role in maximizing IT assets, choosing the right integration technology can be a daunting task, given the number and variety of available options. As shown in Part Three of this book, each integration pattern represents a unique solution set. Matching the business problem to the right integration pattern will help you identify the appropriate technology, making the implementation of the business solution significantly easier. It will also help control implementation costs and reduce risk.
Although each integration pattern has its unique technology requirement, there is also significant overlap across patterns, including translation, transformation and routing services, repositories, adapters, development environments, management, and security. Therefore, although it is helpful on a project level to use the patterns to guide implementation, it is also important at an enterprise level to control redundancy and manage reuse.
The Reference Architecture for the fully integrated enterprise represents an aggregated view of all the integration patterns, into a fully integrated infrastructure (see Figure 14-1, page 264). Companies with multiple integration requirements will inevitably require multiple integration patterns. The purpose of this aggregated view is to minimize redundancy of infrastructure services and maximize reuse and ROI of IT assets.
Although every integration pattern includes translation and transformation, the integrated enterprise reference architecture has only one translation and transformation service. However, every integration pattern discussed in Part III has translation and transformation. Likewise, rules engines may appear in a number of different application patterns, including process automation and composite applications. Ideally, each application will need only one type of adapter; all types of servers could use the same translation and transformation; there would be one repository for all types of metadata, etc.; and all the integration services could be commonly managed by one management console.
This optimized service-oriented integration architecture is not quite possible with the current state of technology. However, companies should strive to minimize redundancy at an enterprise level as much as possible to decrease long-term maintenance costs. This requires a strategic approach to integration.
The fully integrated enterprise can only be achieved through a strategic approach to integration. This statement is made with the full knowledge that the majority of integration projects are tactical implementations. Throughout this book we have attempted to lay out a convincing argument, including supporting case studies, that although the strategic approach requires an initial higher investment, in the long run it significantly reduces the time and cost of implementing new solutions, while increasing business agility. However, the strategic approach requires upper level IT and business management support, as well as a supportive organizational culture.
Although upper level management support is crucial, sometimes an evangelist is required to inspire the commitment. You can use the case studies in this book to help build a compelling argument. If the evangelist route doesn't work in your organization, an initial success can sometimes speak louder than words. A tactical approach can be taken to prove out the underlying concepts and help to convince the organization to move forward strategically.
Even with a mandate to move ahead, a pervasive “not invented here” attitude in the organizational culture could sabotage any potential benefits of the integration infrastructure. Maximizing IT assets requires a culture of reuse. This culture needs to be built both from the top down and the bottom up. From the top down, management needs to invest in resources and processes to maximize reuse, including architects to participate in design reviews to identify opportunities for reuse, and reuse repositories to manage the reusable services. From the bottom up, application developers need to be rewarded for reuse, both when including an existing service in a new implementation, and for contributing reusable services to the repository. Rewards can be a fast and effective way to change behavior and create the desired cultural change.
The relationship between the business and IT organization has a large impact on whether a strategic approach or tactical integration approach is used. The strategic approach requires that there exists a positive working relationship between the IT and business groups. Otherwise, they will not make the initial investment required, even if the tactical approach ends up ultimately costing more. Ironically, a strategic approach helps to build the business and IT relationship. However, without at least some initial business support the strategic approach is difficult to apply.
The goal of this book is to help readers understand the different integration approaches and technologies. It provides tools, including templates and reference architectures, to develop requirements and specifications for guiding the approach to integration on both the strategic enterprise level and on tactical projects. This book is not intended to be a full-blown integration methodology. However, it forms the cornerstone for putting a methodology in place.
Apply these tools within the context of your organization's existing methods, to fill any existing gaps. The templates can be applied as is or merged into the organizations existing methodology.
Integration is not a point in time, but a journey that must be taken by any successful IT organization. Just like networking in the late 1980s and early 1990s, integration will have a lot of uncertainty and challenges. However, over the next decade integration technology will become an essential part of the IT infrastructure and will be expected to work 24/7; require very little effort to setup, use, and operate; enable the creation of business services and processes; and connect these services and processes to any desktop and portable device.
No one can predict all of the right choices. This book lays out the landscape that has emerged, the important problems and patterns, and case studies from best-in-class organizations that are living examples of the importance and benefits of integration. This book is a starting point you can use. Your challenge will be to learn as you go and adapt to the specific needs of your organization.
A good mantra for the enterprise integration journey is “think strategically, act tactically.” While meeting the tactical integration needs of the organization, incrementally build an infrastructure to accommodate the next implementation, and the next. “In the long run, we only hit what we aim at” (Henry David Thoreau). Create an enterprise architecture plan that will serve as your roadmap, and implement it in a tactical fashion, project by project.
We sincerely hope this book will be a helpful guide on your integration journey.