Java CAPS is a toolbox that supports many Enterprise Application Integration (EAI) styles. Its core components deliver message-based application-to-application integration. Java CAPS also enables other styles of integration, including composite applications, Web Services orchestration, single-customer view, and service-oriented architectures.

The fundamental premise of application integration is that it should be noninvasive. This philosophical approach minimizes application changes and costs associated with integration. To that end, it leverages application capabilities rather than requiring applications to adapt to it. Because applications’ integration capabilities vary, so too do integration styles that can be applied to them. In this spirit, an integration solution may use many different integration styles, driven by the capabilities of the applications being integrated. The following sections discuss the major integration styles and identify some of the Java CAPS components that can assist in their implementation.

This chapter also discusses issues of scalability and resilience and Java CAPS facilities that can assist in solution factoring for distribution and replication. These topics are discussed in greater detail in Chapter 13, “Scalability and Resilience.”

File transfer is the most basic integration style. On the inbound side (inbound to the integration solution), it relies on the applications’ ability to write data to files in a file system. On the outbound side (outbound from the integration solution), it relies on the applications’ ability to read and process data from files in the file system. Applications would already be capable of interfacing with the file system or would be extended to produce/consume files. This is a rudimentary, non–real-time, batch-based, point-to-point approach that nevertheless is valid and appropriate in appropriate circumstances. A file transfer–based solution is not, by its nature, very scalable or very robust.

Java CAPS supports the file transfer integration style with a collection of Batch eWay Adapter variants. The Batch Local File eWay is appropriate for reading and writing files in the local file system.

The Batch FTP eWay is appropriate for pulling and pushing files to or from remote FTP servers. It includes support for Secure FTP variants such as FTP over SSL, SFTP (Secure FTP, a subprotocol of SSH), SCP (secure file copy, a subprotocol of SSH), and security infrastructure traversal variants with support for SOCKS and SSH tunneling.

The Batch Record eWay enables payload, delivered through one of the Batch eWay variants, to be broken up into fixed-length or delimited records, as well as assembly of messages into fixed-length or delimited records for writing to a file. The Batch Record eWay can be used in conjunction with the other Batch eWays to implement data streaming solutions that minimize memory consumption when processing large file payloads.

Finally, the Batch Inbound eWay, with its built-in interval times and the unique name feature, facilitates implementation of polling solutions that rename files as they find them and deliver unique filenames and paths to other inbound Batch eWays for processing.

Batch eWays can be used in conjunction with other Java CAPS components to implement file transfer–based solutions.

[EIP] briefly discusses issues with multiple applications sharing a database and concludes that it is not the most appropriate method of application integration. While having multiple applications share one database might be difficult, because of the difficulty in developing a database schema that satisfies requirements of multiple disparate applications, it is no easier to share data between applications each with its own database. Whereas a solution with a single database shared by multiple applications does not require integration, a solution with multiple databases, or with a mix of a database and other external systems, does.

The major advantage of using database-based integration is that applications that already use databases do not need to be modified to enable the integration. Care must be taken, however, when updating data bypassing application logic, to avoid destroying referential integrity, some of which may be enforced by the databases and some of which may be provided by application logic.

The Java CAPS suite includes a series of eWay Adapters developed specifically to natively interoperate with all major relational databases and, through JDBC, with all JDBC-compliant databases. These adapters encapsulate all common database interaction functionality, including insert, update, delete and select operations on tables, views, prepared statements, and stored procedures. Each database-specific adapter provides a wizard that enables creation of Object Type Definitions corresponding to database objects, using schema metadata directly from the database. Solutions that select data from one or more databases and/or populate or update one or more other databases can be readily developed. Using Java-based database triggers, where a particular database supports this feature, a Java CAPS–based solution can be triggered to process inserted or updated data or to synchronize deletion with some other external system or data store, as close to real time as possible. Where not possible, a Java CAPS solution can poll database tables or views or schedule invocation of stored procedures.

For a database data structure to be accessed, an integration solution must embed the knowledge of this structure. This makes the solution tightly coupled to database structures and therefore potentially brittle. To offset this, a message transformation component may serve as a wrapper for the underlying database object, providing indirection and isolating the rest of the integration solution from the database structure.

Because of the heavy overhead of accessing and manipulating database resources, solutions that access databases may not be as scalable or perform as well as solutions that do not. If database schema contains binary large objects (BLOBs) or character large objects (CLOBs), the solutions will require special handling and will be more complex to implement and more resource intensive.

Distribution of functionality among multiple independent processes or hosts is what differentiates a monolithic application from a distributed application. Remote procedure call (RPC) [EIP] is the mechanism through which an application can remotely invoke specific functionality (“procedure”) exposed by another application. This implies that the procedure encapsulates some functionality of interest, that the interface is published, and that both applications use the same wire protocol and data representation and semantics.

Of the common RPC technologies, Java CAPS, being Java-based, naturally supports Java Remote Method Invocation (RMI) and Common Object Request Broker Architecture (CORBA), though low-level Java programming is required. Through its ability to consume and publish Web Services, it also naturally supports that technology, though it does not support the RPC/Encoded style, as RCP/Encoded style is not WS-Interoperability compliant. In addition to these built-in capabilities, Java CAPS provides a COM/DCOM eWay Adapter that allows incorporation of COM/DCOM resources, such as Microsoft Excel spreadsheets, into enterprise integration solutions.

When necessary and appropriate, RPC can be used to orchestrate remote resources or expose a Java CAPS solution as a remotely invocable resource. Use of RPC-based technologies typically results in tightly coupled solutions, so care must be taken to minimize the amount of remote application knowledge that is built into a Java CAPS solution. Also note that most of the RPC technologies are platform-specific (COM/DCOM, Java RMI), thus making solutions nonportable and difficult to deploy and test. Care must be taken to isolate the use of such technologies into small, independently deployable components.

[EIP] discusses shortcomings of file transfer, database sharing, and RPC and concludes that asynchronous messaging is the most appropriate pragmatic approach to EAI. [EIP] elaborates on the advantages and disadvantages, and contrasts this style with other integration styles.

Java CAPS is an excellent candidate to build a messaging system. It is, fundamentally, a many-to-many middleware messaging system, suitable for implementing both event-driven and service-oriented architectures. As a backbone messaging infrastructure, Java CAPS supports a number of Java Message Service (JMS) implementations from both Sun Microsystems and third parties. Both point-to-point and publish/subscribe messaging are supported. JMS API Kit is available to interface external applications directly to the JMS infrastructure. Message transformation logic can be implemented both in Java (Java Collaborations), Extensible Style Language (XSLT Collaborations), and Business Process Execution Language (BPEL). Message routing can be implemented in Java and BPEL. Over 80 eWay Adapters are available to receive messages from external sources and deliver messages to external destinations. An eWay Development Kit facilitates development of custom adapters, if required. For these reasons, and because of its distributed nature, the product provides an enterprise architect with the flexibility to design the most appropriate solution to a given integration problem.

The notion of building services, with published interfaces and using standard wire protocols to encapsulate and expose specific application functionality, is gaining increasing popularity among enterprise architects. The idea that services can be combined in different ways to rapidly deliver differing business functionality has led to a great deal of activity aimed at the development of service-oriented architectures. Web Services, an implementation of the service concept using HTTP as the wire protocol to facilitate remote invocation across corporate firewalls, adopted XML for data representation and are generally invoked synchronously.

Business functionality is delivered, in this approach, through sequential invocation of appropriate services, also called service orchestration. Java CAPS eInsight is a business process manager and service orchestration engine. It can invoke Web Services, access non–Web Services external resources through eWay Adapters, implement transformation and routing logic, and so become the technology foundation of the enterprise service–oriented architecture. eInsight Business Processes and Java Collaborations can be exposed as Web Services invocable from other eInsight Business Processes, Java Collaborations, or external clients.

It is worth pointing out that Web Services—Simple Object Access Protocol (SOAP) over HTTP—largely suffers from the same problems as the RPC, most notably reliability, availability, scalability, latency, transactional integrity, ownership and control, and tight coupling between the client and the service. Deployment of Web Services intended to operate over untrusted networks, like the Internet, adds the issue of security and exacerbates the problems of latency, ownership and control, availability, and trust. These problems resulted in an explosion of standardization activities aimed at providing, for Web Services, the qualities of service taken for granted in traditional monolithic applications and which the original designers of SOAP never built into their design.

A complementary notion, that of building end-user applications providing a Web-based interface over orchestrations of enterprise resources, is the notion of Composite Applications. It is argued that 80 percent of functionality required by a given end-user application would already be available as services or external system interfaces that can be exposed as services. The user interface and a small amount of “integration glue” would be required to deliver a new composite application that leverages the existing investment. Java CAPS provides all the tools necessary to build composite applications. The very name of the product embeds the notion of composite applications. eVision, a part of the Java CAPS suite, is the tool used to develop Web pages and Web page flows that orchestrate arbitrary resources in the same manner as a regular business process would.

Java CAPS allows the enterprise architect the flexibility to use all of the architectural styles in the same solution if requirements of the business dictate it.

A typical integration solution connects multiple external systems to each other, performing enrichment, transformation, auditing, and other activities as messages are processed. The complexity of work that the integration solution must perform, and the architecture of the EAI toolset to be used, largely dictates the number of discrete components involved. The number of components, in turn, dictates both the scalability and the resiliency options available to the solution architect. Scalability and resiliency are both important, nonfunctional goals of the EAI architecture.

The fundamental component of the Java CAPS EAI solution is the Integration Server. Out of the box, Java CAPS 5.1.x Integration Server is the Sun Application Server 8.0 Platform Edition. Because they are Java EE compliant, Java CAPS Enterprise Applications can be deployed to a number of other Application Servers, including the Sun Application Server 8.2 Enterprise Edition and BEA WebLogic 9.1.

Java CAPS integration solutions are collections of Java EE applications, externally manifested as Enterprise Application Archive (EAR) files—self-contained units of functionality. Each Java EE application, the EAR file, is deployed to an Integration Server in its totality: no Java EE application can span multiple servers. The same EAR file can be deployed to multiple Integration Servers concurrently if it makes sense and if potential external resource conflicts are avoided or resolved. Thus the EAR file is the smallest unit that can be distributed among physical resources.

In Java CAPS, an EAR files is generated as a result of a developer creating a deployment profile, assigning components to logical host containers, and building the resulting integration solution project.

Decisions as to how many Java CAPS logical hosts and deployment profiles to use in an EAI solution and what components to assign to each deployment profile in each logical host are the most critical decisions affecting granularity of distribution and therefore scalability and, potentially, resilience.

Note

In Java CAPS, an EAR file is generated as a result of a developer creating a deployment profile, assigning components to logical containers, and building the resulting Enterprise Application. A deployment profile maps logical components to external system containers within a Java CAPS environment. An environment is a collection of external systems, Integration Servers, and, optionally, Message Servers. What Java CAPS components can be mapped in a single deployment profile is determined by what components are configured in one or more connectivity maps. A single deployment profile can use components form one or more connectivity maps, as long as all the connectivity maps are in the same project or subproject as the deployment profile. A single deployment profile can map components to multiple logical hosts, each with its own Integration Server and/or Message Server.

Note

Java CAPS EAI solutions are generally event driven. An external event triggers a component to receive a message from an external system or poll for a message in the external environment. The JCA Adapters, or eWays, as well as other endpoints like JMS Clients, are used by Java Collaborations or eInsight Business Processes. The eWays cannot be separated from the Java Collaboration Definitions (JCDs) or Business Processes that use them. Logical break points for dividing Java CAPS EAI solutions into separate deployments are JMS Destinations. Since each JMS Destination name is global to the JMS Message Server to which it is deployed, a connectivity map that contains a JMS Destination called queueA will be referring to the same JMS Destination as another connectivity map that contains a JMS Destination object called queueA, so long as both are deployed to the same Message Server.

By deciding to deploy all components of the EAI solution through a single deployment profile to a single logical host, the solution architect creates a centralized EAI infrastructure—a hub-and-spokes model of old. If the solution is small and there are no resilience considerations to address, this model is perfectly valid. Thanks to the inherent multithreading of the J2EE environment, this solution is much more scalable, within the single Integration Server, than the traditional hub-and-spokes solutions were.

If the ability to distribute components over multiple platforms for load-balancing, resilience, or scalability is desired, the solution must be architected to consist of multiple Enterprise Applications (in separate EAR files), each implementing a specific part of the overall solution. Solution components would exchange messages using JMS infrastructure or other distributed communication models like Web Services.

In strictly eGate-based solutions, where business functionality is implemented exclusively using Java Collaborations, breaking up the solution into deployable components is relatively easy. The enterprise architect merely picks a series of components that communicate with some other series of components using common JMS Destinations, and assigns them through one or more connectivity maps and one or more deployment profiles to multiple Integration Servers and Message Servers.

In solutions where most of the business functionality is implemented using eInsight Business Processes the refactoring task is harder. Distributed architecture must be devised before or concurrently with the design of Business Processes. The reason is simple: each Business Process is a single, indivisible component, deployed in its totality in a single Integration Server. There is typically a conflict between dictates of solution distribution and dictates of business process completeness and visibility. The architect must carefully weigh the benefits of implementing complete, large business processes against the costs of component failure and performance implications. Conversely, breaking up processes for performance or fault-tolerance reasons will result in loss of overall complete process visibility.

Ultimately, a solution architect must consider all the tradeoffs and design a solution that best reflects the business and technical environment on a case-by-case basis.

Scalability is the ability of the system to process increasing amounts of work by replicating components. Resilience is the ability of the system to continue operating in the face of failures.

While inherent scalability and resilience features are present in the J2EE platform, the Application Server, JMS, and the hardware platforms, scalability and resilience of the solution must be architected into it from inception to take advantage of these inherent features and add explicit features where they are lacking.

One of the fundamental means to facilitate scalability and resilience is factoring of the solution into smaller units for distribution and replication. Scalability and resilience requirements of the solution will initially dictate decisions about factoring. These decisions must, however, be informed by the capabilities of the toolset and the physical environment.

Java CAPS solution factoring is dependent on the nature of the solution. Strictly eGate-based solutions would be typically broken up at JMS Destinations, which are the logical break points. Strictly eInsight-based solutions may not have many JMS Destinations, so large business processes may need to be explicitly broken up into multiple smaller processes to address resilience and scalability requirements.

Technical details of solution factoring are discussed at some length and illustrated with examples in Chapter 13.

Breaking up a large solution into components using JMS Destinations as logical break points creates an opportunity to leverage JMS for load-balancing and scalability. A JMS queue can have multiple Competing Consumers, components that receive messages from the same queue, each of which will get a message if there is one. A JMS Destination–based load-balancing configuration comes into being when components of the solution, which receive messages from a specific queue, are replicated and multiple copies are deployed to multiple Integration Servers. In this configuration, the JMS Message Server is a single point of failure, so while scalability requirement can be addressed this way, resilience cannot. To also address this resilience issue, the JMS Message Server would have to be deployed in a clustered configuration with JMS backing store deployed to cluster-shared storage, or else a JMS Grid–based fault-tolerant JMS solution would be needed. This topic is further discussed in Chapter 13, section 13.5.

This chapter discussed the major integration styles and identified some of the Java CAPS components that can assist in their implementation. Integration styles, including file transfer, database sharing, remote procedure invocation, messaging, and service orchestration, were briefly discussed.

This chapter also discussed issues of scalability and resilience and Java CAPS facilities that can assist in solution factoring for distribution and replication.