program: n. A magic spell cast over a computer allowing it to turn one's input into error messages. v. tr. To engage in a pastime similar to banging one's head against a wall, but with fewer opportunities for reward. | ||
| --Unknown | ||
As we conclude, let's look at the state of the art in EAI technology, as it exists now and where it is likely to be going in the next few years. If you're considering an EAI approach in your organization, now might be the time to "lay your cards on the table" and solidify some decisions.
EAI is clearly a complex problem. However, as with any complex problem, once it is broken down to its component parts, the solution becomes simply the aggregation of a number of solution sets. In this case, the solution sets include a combination of a variety of approaches and several types of technology.
As in the world of technology at large, the world of EAI is changing rapidly. The problem that EAI seeks to solve is morphing from the very simple to the very complex, even as it moves from a departmental problem to an enterprise-wide problem. As a result, few companies have been able to get ahead of the "EAI curve." They have yet to discover the full potential of EAI. As the problem grows, so does the potential benefit to the solution. So the technology continues to respond. The nirvana of EAI has yet to be created. In short, there's a great deal of work ahead of us.
As EAI evolves, the problem domains are changing. No sooner is a "traditional" EAI problem solved (such as application-to-application and database-to-database integration), then EAI expertise and technology is being applied to more complex, but more rewarding, business issues.
This is both natural and smart. As systems are integrated within the enterprise, it is just good business to take that experience and technology and apply it to other opportunities that influence the bottom line.
Although supply chain integration, or Inter-Enterprise Application Integration, was covered earlier in Chapter 16, it's important to recall here that supply chain integration is an old issue, one well served by new EAI tricks. Using familiar approaches and technologies, it is possible to bring together very different systems that exist in different companies and allow these systems to share information easily.
Using automation to integrate the supply chain is nothing new. However, the application of new technology and standards, such as message brokers and XML, provides many additional opportunities to address this challenge effectively and efficiently (see Figure 20.1). What's surprising is that most EAI vendors don't seem to share this vision. They are still concentrating on enterprise integration, satisfied to solve most external integration problems by exposing systems with Web servers. While Web integration is always critical, the heavy lifting occurs at the back-end, where system binding ultimately needs to occur.
EDI (Electronic Data Interchange) represents a sound solution, but its complexity and expense have largely doomed it—although EDI will remain a point-of-integration for some time. XML provides a much more simplistic approach and, as such, is less expensive to employ. However, many organizations have chosen not to use either XML or EDI. They have gone with some other message standard (e.g., JMS) or a proprietary communications mechanism. As a result, a hodgepodge of technologies and standards will remain in place even as we seek to address the supply chain integration problem.
Ultimately, you need to select a middleware product to move this information. Although message brokers seem to provide the most flexible solution, in many instances transactional middleware (e.g., application servers and TP monitors) is the better fit. Middleware is sold primarily for enterprise integration, but the application of this technology to supply chain integration is also occurring and will grow in importance.
Another clear trend is the movement from data-level to application-level (application interface–level and method-level EAI) integration. As noted previously, data-level integration provides an inexpensive mechanism to integrate applications because, in most instances, there is no need to change the applications. While data-level integration provides a rudimentary solution for most EAI problem domains, the integration of both application services and application methods generally provides more value in the long run. The downside, at least with method level, is that it is necessary to change the source and target applications or even, in many instances, create a new application (composite applications).
This approach is consistent with the "baby step" approach most enterprises take in implementing solutions to integration problems. Often EAI solutions are created in a series of small, low-risk steps. This type of implementation works from the department to enterprise but never from the enterprise to department. Data-level EAI provides most organizations with the opportunity to get EAI under control, with low risk, then to plot a strategy to move to application-level integration, as more time and money become available and the appetite for risk increases.
Unfortunately, any discussion of EAI tends to leave a number of "loose ends"—issues that don't really fit into the context of any other discussion of the EAI solution but that need to be considered before the subject can be closed. These issues include: security, performance, and administration.
Too often, security seems to be an afterthought to the implementation of new technology. In order to address security properly, it is necessary to build the system or, in this case, the integration solution from the ground up. While a detailed discussion of security is beyond the scope of this book, it's important to remember that whenever information is sent or received from enterprise systems, when interfaces to the systems are built, or when middleware is being implemented, security must be considered. The vulnerabilities of any enterprise remain only a handful of mouse clicks away—a very sobering thought when money and time are on the line. A few mouse clicks and an enterprise's most valuable asset, its information, could be laid bare to the predatory desires of a competitor.
Security must be considered early in the EAI project process. In most cases, EAI security will be built on top of an already existing security structure within the source and target applications (e.g., Top Secret, RACF, or Windows NT security) to be integrated. Therefore, in addition to integrating applications, EAI will need to integrate the security systems.
Just as with security, performance issues must be part of the EAI design from the ground up to insure that a set of integrated systems is going to perform well. Many organizations, sensitive to the security risks of integration, do consider security during the EAI development process. However, many of these same organizations fail to adequately address performance until it is too late.
Such complex issues as message rates, transactions-per-second, and interface performance must be taken into account when considering performance engineering and EAI. The most reliable approach to performance in the design of an integration solution is to select the technology and then do a simulation model to make an educated assessment of how the solution will perform. Using simulation tools, it is possible to determine the time it will take for a message or a transaction to move between applications. This simulation must test the various entities of the system at the component (e.g., server) and system level (e.g., integrated systems) to ensure the overall performance of the EAI solution (see Figure 20.2). Just as a chain is no stronger than its weakest link, an integration solution is no more efficient than its slowest performing component. Testing identifies problem components before the solution is actually implemented.
The administration of the EAI solution must also be considered. Within typical EAI problem domains application servers, message brokers, TP monitors, and even new composite applications are being implemented to integrate the enterprise. These all represent new technology that must be maintained. Such maintenance includes performance management, disaster recovery (backup and restore), security administration, and configuration management. Over time, in most problem domains, more money will be budgeted for administration than for creating the original solution. For this reason, creating an EAI solution with an eye to administrative efficiency is in the best interest of the bottom line. Of course, solutions that are easy to administer generally provide more up time and a better user experience.
While the majority of our attention has been devoted to various approaches to EAI, you'll discover as you shop for solutions that each EAI vendor has its own take on what EAI is. Who's right? Who's wrong? Unfortunately, as with everything else in the EAI domain, the answer is not black and white.
EAI is a combination of problems, and each organization has its own set of integration issues to address. As a result, it is virtually impossible to find a single technological solution set that can be applied universally. A particular enterprise's EAI solution will generally require products from several different vendors. At this time, and in the foreseeable future, one-step shopping is simply not an EAI reality.
Although vendors' approaches to EAI vary considerably, it is possible to create some general categories. These include:
Data oriented
Application integration oriented
Process automation oriented
Transaction oriented
Distributed object oriented
You should note that these vendor solutions are independent of the levels of EAI (database, method, application interface, and user interface), and may inhabit one, two, or three of the levels of EAI.
Vendors that promote the data-oriented approach to EAI make the case that integration should occur between the databases—that is, databases should be viewed as the primary points of integration; thus they believe in data-level EAI. Even within data-level EAI, there are many approaches, and each vendor is quick to promote its particular solution. Data-level solutions can be categorized into two categories: data replication and data federation.
Data replication means moving data between two or more databases, databases that can come from the same vendor or many vendors, or even databases that employ different models. The trick with database replication is to account for the differences between database models and database schemas by providing the infrastructure to exchange data. Such solutions are plentiful and inexpensive. Most relational database vendors, including Sybase and Oracle, provide database replication services within their own product offering. These replication engines typically exist within the database engines, at either end (see Figure 20.3).
Many database-oriented middleware solutions are on the market that provide database replication services as well. This is accomplished by placing a layer of software between two or more databases. On one side, the data is extracted from the source database or databases, and on the other side, the data is placed in the target database or databases. Many of these solutions provide transformation services as well—the ability to adjust the schemas and the content so they make sense to the target database (see Figure 20.4).
The primary advantages of database replication are simplicity and low cost. Database replication is easy to implement, and the technology is cheap to purchase and install. Unfortunately, data replication loses value if methods need to be bound to the data or if methods are shared along with the data. To accomplish these things, method sharing solutions such as transaction-oriented or application integration–oriented EAI must be considered (see the discussion of these solutions later in this chapter).
Although database federation has been around for awhile, the solution set has been perfected only recently. Database federation is the integration of multiple databases and database models into a single, unified view of the databases (see Figure 20.5). Put another way, database federations are virtual enterprise databases that are comprised of many real physical databases.
Database federation software places a layer of software (middleware) between the physical distributed databases and the applications that will be viewing the data. This layer connects to the back-end databases using available interfaces and maps the physical databases to a virtual database model that exists only in the software. The application uses this virtual database to access the required information, and the database federation handles the collection and distribution of the data as needed to the physical databases.
Figure 20.5. Data federation software is middleware that allows an application to view a number of databases through a single view.
The advantage of using database federation software is the ability to bind many different data types into one unified enterprise-wide model.
Database federation allows access to any connected database in the enterprise through a single well-defined interface. This nails the data-level EAI problem, providing the most elegant solution. While this solution, unlike replication, does not require changes to the source or target applications, changes do have to be made at the application level to support federated database software. This is due to the fact that different interfaces are being used to access a different database model (the virtual database).
Application integration–oriented product solutions focus on the integration of both packaged and custom applications by using well-defined application interfaces, and supports the data, method, application interface, and user interface EAI levels. Right now, the interest in integrating popular ERP applications (e.g., SAP, PeopleSoft, and Baan) has made this the most exciting EAI sector. (While distributed object and transaction-oriented solutions may be applied to this space—because here you program your way to success—message brokers vendors are promoting their products as the preferred solution.)
Message brokers support application integration–oriented solutions by connecting into as many custom or packaged applications as possible through adapters. They also connect into technology solutions that include middleware and screen scrapers as points of integration (see Figure 20.6).
The advantage of using application integration–oriented products is the efficient integration of many different types of applications. In just a matter days, it is possible to connect a SAP R/3 application to a Baan application, with the application integration–oriented solution accounting for the differences between schema, content, and application semantics by translating on the fly the information moving between the systems. Moreover, this type of solution can be used as a database replication solution, which is also able to connect to application interfaces.
The downside to using application interface–oriented products is that there is little regard for business logic and methods that may exist within the source or target systems, logic and methods that may be relevant to a particular integration effort. In such a case, transaction- or distributed object–oriented solutions (composite applications or a pure method-level approach) are probably a better choice. Ultimately, application interface–oriented technology will learn to share methods as well as information, perhaps by joining forces with transaction-oriented or distributed object–oriented solutions. However, for now you will have to make an either/or decision.
Process automation–oriented products, as covered in the previous chapter, are those solutions that layer a set of easily defined and centrally managed processes, or workflow solutions, on top of an existing set of processes contained within a set of enterprise applications (see Figure 20.7).
Figure 20.7. Process automation–oriented products layer a set of centrally managed processes on top of an existing set of enterprise processes.
The focus is to bring together any relevant enterprise processes to obtain the maximum amount of value while supporting the flow of information and logic between these processes. These products view the middleware or the plumbing as a commodity and provide easy-to-use visual interfaces for binding these processes together.
In reality, process automation is another layer of value on top of existing EAI solutions, solutions that include message brokers, application servers, distributed objects, and other middleware layers. They offer a mechanism to bind disparate processes together and to create workflow solutions that automate tasks once performed manually. However, by diminishing the importance of the plumbing, they miss the larger picture. In reality, no single EAI vendor has solved the plumbing issues. Ultimately, the solution to these issues will be delivered by a combination of process automation and middleware vendors. That being the case, it is clear that the binding of middleware and process automation tools represents the future of EAI.
Transaction-oriented products are those middleware products that use the notion of a transaction, with connectors to back-end systems, to integrate applications, a clear method-level EAI approach. Examples of these products include TP monitors and application services. Although transaction-oriented products can be used for data integration, method-level integration is where these products shine. With these products, it is possible to create common methods (transactions, really) and share those methods among many connected applications (see Figure 20.8).
The advantages of transaction-oriented products are scalability and reliability. However, success with these products requires a tremendous amount of coding. As time marches on, transaction-oriented middleware will combine forces with messaging and message brokers, providing EAI architects and developers with the best of method-level, data-level, and application interface–level EAI.
Like transaction-oriented products that exist at the method level, distributed objects (such as those based on the COM or CORBA standards) also provide EAI architects and developers with an opportunity to share methods. The elegance of the distributed object architecture, the built-in communications, and the ability to support the distributed model has led many to believe that this is the ultimate EAI technology. However, distributed objects have a long way to go before they can support most large-scale EAI projects. They still fall short when it comes to supporting transactionality, messaging, and easy-to-use tools for building and deploying distributed object EAI solutions. A great deal of custom coding will be required to leverage the power of distributed objects within an EAI solution.
It's important to note that the distributed object standards are moving in the right direction by including built-in support for transaction and integration with Web standards such as Java. Still, it will take some time and momentum in order for distributed objects and EAI to fit together. At this time, it doesn't appear that distributed objects have the luxury of either.
Although these technologies represent different approaches to EAI, most EAI problem domains will discover that they use most levels of EAI (method, application interface, data, and user interface) when integrating an enterprise. Therefore it is important to select the appropriate technology that works at each level. Wishing for a single technology and a single vendor for all levels is just that—wishing.
Developers and architects may pass information between databases, supporting data-level EAI, using any number of tools. These include newer message brokers, as well as more traditional data replication software. At the application interface level, message brokers seem to do a better job connecting to and moving information in and out of packaged applications. Moreover, at the user interface level, message brokers seem to provide a better event-driven approach, including connectors to most traditional screen-scraping technology. That said, it's in implementing method-level EAI, or creating composite applications, where all message brokers fall short.
Message brokers are not designed to house or share application logic. Rather, they provide rudimentary rules engines to support operations such as the identification, transformation, and routing of messages to the appropriate systems. If this is the type of EAI required, then application servers are the correct choice, providing the ability to integrate many different systems by sharing common business logic and, thus, information.
It should be clear that message brokers and application servers are complementary and converging. While message brokers do a more-than-adequate job in providing event-driven, asynchronous access to many different types of systems, application servers do a much better job in providing the infrastructure to share common logic. Both technologies integrate the enterprise—solve the EAI problem—but do so in very different ways. Although both application servers and message brokers use the middle-tier integration approach, application servers are front-end and are application development focused where message brokers are back-end and operations and process oriented.
Application server and message broker vendors believe the adage, "If you can't beat 'em, join 'em." That's precisely where the application server and message broker vendors are going. Most message broker vendors are partnering with application server vendors to ensure that, at least in the short term, they'll have a solution at the method level. The end result of these partnerships is sure to be the creation of hybrid products that support both application server and message-brokering features. The speed and approach will vary greatly from vendor to vendor, with larger message broker vendors purchasing smaller, more vulnerable application servers and larger application server companies consuming the smaller message broker players. At the same time, some application server vendors will incorporate message-brokering features in their products, and some message broker vendors will learn to provide products more like application servers.
The integration of application servers and message brokers results in a hybrid "super" product that may be called an "application and integration server." This product will offer a location for shared application logic and the ability to build composite applications (see Figure 20.9), as well as the ability to access any system using a synchronous or asynchronous communications model. Moreover, this new product will be able to account for the difference in application semantics, schema, and content.
A discussion of the existing technology is comparatively simple when viewed against the future. Here, reason, guesswork, and a good crystal ball have to suffice. First we must consider approach. There are two elements that must be improved within the enterprise: architecture and application design.
Anyone who suggests that enterprise architecture is a lost art is just plain wrong. In order for it to be "lost," it would have had to be "found" once, and that has never been the case. Within most enterprises, technology has been implemented for years without considering how it would all someday have to fit together. Ultimately, this is the source of the EAI problem—bad architecture. Many organizations have a problem with architecture because it requires discipline. A technology and business plan has to be created and adhered to, avoiding the temptations of using new and more trendy technology unless it is able to prove its value within the enterprise.
The new interest in architecture is led by the newly conceived role of architect, which now exists within many enterprises. Today, the architect coordinates the installation and configuration of technology, assuring that the enterprise is using the best solution to add the most value to the business. Enterprise architects also manage the network and systems, and the integration of all technologies, typically by leveraging an enterprise architecture or master plan.
Unfortunately, the simple fact is that, within most enterprises, the damage has already been done. Indeed, the enterprise architects are going to be in repair mode for years. EAI is only one of the many problems they will be called upon to solve, finally bringing together applications that were never designed to communicate information in the first place. Given the importance of the solution, the interest in enterprise architecture will clearly grow over time.
If architecture is an enterprise issue, then application design is certainly a system-level issue. Although applications have been designed well for years, the failure has been with applications not designed to interact with other applications. In the future, application architects will pay more attention to designing and building the mechanisms that are necessary to move information in and out of an application.
Middleware is only one part of the application integration story. While middleware does a good job moving information from one application to the next, the ability to externalize both processes and information from any application comes with its own set of problems as well as opportunities. The name for this concept varies from organization to organization, but the term "application externalization" is the most descriptive. This is the process of changing or enabling an existing application so that other layers of technology can take advantage of it.
At issue is the fact that most applications are not built to work with other applications or technology, and thus are not able to externalize both processes and data encapsulated within a given application. While we've traditionally focused on middleware and the technology employed to connect applications together, it may be just as important to focus on the science of enabling these applications, both new and old, to communicate outside of themselves.
EAI would not be so important now if the "ball" had not been dropped in the past. As technology paradigms shifted over the years, applications were built around the new paradigm. These shifts included centralized, client/server, and distributed computing, Web-enabled applications, and packaged applications. Now, all these applications are in place, and we have discovered that integrating logic and data is a daunting proposition. Enter EAI, along with an opportunity to finally integrate all of these disparate systems with a minimum impact on the applications and the way an enterprise does business. Unfortunately, the approaches and technology behind EAI are still in their infancy. The knowledge necessary to fully integrate an enterprise is lacking. Indeed, as it functions now, EAI tends to be a stopgap measure to avoid creating new systems. The ultimate goal of EAI is to bind all enterprise systems together in such a way that any application can access any method or any piece of data without delay. It will take some time before this goal is achieved, but don't fear, we'll get there some day!