This is the first in a series of chapters that specifically address [EIP] concepts and patterns. We address a number of [EIP] patterns and discuss how Java CAPS can be used to implement specific message-related patterns, which [EIP] groups together under the heading of Message Construction. In particular, the following patterns and specific Java CAPS concepts and solutions illustrations are discussed:

[EIP] discusses the subtle differences between a Command Message, a Document Message, and an Event Message, pointing out that the differences are semantic rather than structural or technological and that the semantics are solution-specific. From the Java CAPS perspective, any message is a Document Message in that it carries business data from component to component. Technologies used to create and manipulate the message are the same regardless of whether additional meaning is superimposed by the application.

It is implicitly assumed that a Document Message has some intrinsic business-specific structure. This structure can be parsed to obtain access to specific information of interest—HL7 Message, Database Table, and SAP IDOC. Chapter 7, “Message Construction,” discusses Java CAPS facilities used for creation of messages and manipulation of their content.

A Command Message [EIP] is a message whose role is to invoke a predefined activity, implemented by a component which is the destination of the message. In Java CAPS, every message would be a Command Message according to this definition. If a distinction is needed, then perhaps a message that causes the component to invoke logic different from the logic used to process ordinary messages should be considered a Command Message. All other messages, from the perspective of this component, would be Document Messages.

Java CAPS solutions, or solution fragments, like Java Collaborations or business processes, are always triggered by a message—a message delivered to an endpoint or a message delivered as a result of timer expiration.

The Scheduler Connector, which manages time schedules, could be considered a generator of Command Messages. The scheduler triggers invocation of the collaboration or the business process with which it is associated when a scheduled time is reached. It sends a message containing a fixed literal string to the component’s listener. This message is a Command Message in that it causes the target collaboration or the business process to be activated and to perform its programmed activities. The component triggered this way typically operates on messages it collects by polling external systems, which is the reason it is designed to be triggered by a timer.

When not triggered by a scheduler, the Java CAPS component is triggered by a message, whose content typically contains business data to be processed. Unlike eWays, which operate on messages with eWay-specific and application-specific structure, JMS endpoints deliver messages regardless of their intrinsic structure. Further, JMS messages contain both the payload and a series of predefined properties, which can be supplemented by user-defined properties. A JMS message payload or properties could be used to construct a Command Message rather then a regular message with business payload.

Event Message in a messaging system is a Document Message. As [EIP] states, “the difference . . . is a matter of timing and content.” An Event Message typically carries very little or no content. Its purpose is to cause the receiver to begin its programmed activity. In this spirit, a Scheduler eWay, Java CAPS–provided timer-based trigger generates Event Messages whenever its timer expires or its schedule falls due. The Scheduler eWay delivers an Event Message, with a fixed body, to the component that uses it. Chapter 2, section 2.2, “The Event Message Using Scheduler,” in Part II (located on the accompanying CD-ROM), illustrates the point with a simple Java CAPS project.

Note

The scheduler configuration is not particularly sophisticated, and the content on the input message is the literal string “StaticString.” Note, too, that the schedule to which the eWay operates is configured through the connectivity map. A change to the schedule requires the solution to be rebuilt and redeployed.

When required to implement a solution where multiple events must take place according to different schedules, multiple Scheduler eWays must be configured. With only a few schedules, this is perhaps not too much of an issue. When the number of different schedules grows, the issue rapidly becomes a major inconvenience.

Cron, of the Unix world, is the typical yardstick used to measure scheduling solutions. Cron-like scheduling sophistication is what is sought. A reasonably sophisticated scheduling solution would support event triggering expressions like these:

It would also have the ability to supply job identifiers and job parameters so that correct jobs can be triggered and variable data provided to them.

When faced with a need for sophisticated scheduling, the first instinct is to build a sophisticated scheduler into the solution under construction. In Java CAPS, that would mean some form of a J2EE container–friendly schedule management mechanism. While a number of Java-based, Cron-like scheduling solutions have been developed, none have been integrated into Java CAPS.

A much simpler way to provide more sophisticated scheduling capability would be to use an external scheduler, perhaps Unix native Cron or Windows native Task Scheduler, to trigger a Java CAPS solution via an endpoint. The endpoint could be a JMS destination or a listener eWay—perhaps a TCP/IP eWay, a HTTP eWay, or even a Batch Local File eWay. The Java CAPS solution need not be aware of scheduling at all. All it needs to do is process a message when it comes—now. The external scheduling solution would decide when the “now” is. Chapter 2, section 2.2.1, “External Scheduler Example,” in Part II, implements a simple project that illustrates this concept.

The obvious advantages of using external scheduler are:

The obvious disadvantage is that the external scheduling solution is external to the Java CAPS solution.

If we consider replacing a number of Scheduler eWays with the same number of TCP/IP eWays to be an issue, we could consider introducing a single TCP/IP eWay that receives all scheduled event messages and uses a fixed or a dynamic router to route these messages to the parts of the solution they are to trigger.

Somewhere between the simplistic Scheduler eWay-based solution and the completely externalized solution is a “heartbeat”-based solution implemented using Java CAPS. One could design a solution that has a single Scheduler eWay that works to a repeated “heartbeat” schedule, say 1 minute. Every minute, a message is injected into the solution to trigger a “Schedule Keeper” collaboration. This collaboration, when it starts, prepares a list of minutes, with a list of jobs, possibly empty, associated with every minute, read from a file or a database. When triggered, the Schedule Keeper collaboration consults the list of minutes and, for the current minute, sends a JMS message for every job that is to be triggered at that time. The message contains job specifications for the job. Some downstream component receives the message and executes its programmed functionality. The Schedule Keeper collaboration would need to have sophistication necessary to parse job specifications, parse time intervals, parse execution windows and frequencies, parse job parameters, and assemble minute-by-minute job lists. This kind of sophistication calls for a reasonable amount of Java code, so it is not provided as an example.

Some messaging solutions require that the recipient is active at the time of send, thereby guaranteeing to the sender that the message was received by the recipient. BEA MessageQ, a proprietary system from Oracle, is one such system. Other messaging solutions operate on a fire-and-forget or a store-and-forward basis, expecting the messaging infrastructure to deliver each message to the intended recipient whether that recipient is active at the time of send or not. JMS is one such system. The major difference, from the architectural perspective, is the timing. In the former case a message is delivered “immediately” or fails “immediately,” so the sender can branch as appropriate upon sending the message. In the latter case, the message is delivered to the messaging system, which “immediately” acknowledges that it has assumed responsibility for delivery to the ultimate recipient. That delivery, however, may take some time if the recipient is not active for some time or may not take place at all if the recipient never appears. The sender will never know what the ultimate outcome was.

In some situations a solution may be architected such that major pieces of functionality are built as functions, modules, or services, accepting some input message, performing some processing, and producing some result. To implement such components, it is necessary that the invocation mechanism allows the invoker to invoke a component, provide the input data it needs, wait for the execution to complete, and receive the result. This interaction is what [EIP] calls the Request/Reply pattern.

In general, you could implement a Request/Reply pattern using any endpoints, with or without messaging infrastructure, as long as the requestor was engineered to make a request, wait for the reply, and receive it. Even file exchange can support the Request/Reply pattern. In fact, the Australian Energy Industry–developed HokeyPokey protocol uses the File Transfer Protocol (FTP) to implement a Request/Reply pattern for submitting aseXML (Australian Standard for Energy XML) documents. The sending component places a file containing an XML document in a “mailbox” of an Energy Hub, then polls the “outbox” at the Energy Hub for an Acknowledgment file. The exchange is not complete until the Acknowledgment file is received or a timeout occurs. This is a classic Request/Reply implementation.

Java CAPS provides a number of mechanisms to implement the Request/Reply pattern. Which of the different mechanisms is appropriate will depend on the problem that needs to be solved. While any endpoint type can be used to implement the Request/Reply pattern, we discuss only the more common, useful, and interesting mechanisms: JMS Request/Reply, HTTP Request/Reply, SOAP Request/Reply, and Web Services Invocation.

JMS Request/Reply

JMS is typically used to build store-and-forward messaging solutions. It also supports implementation of Request/Reply solutions using temporary JMS Destinations. In Java CAPS, a Java Collaboration can be both a requester and a responder in a Request/Reply configuration. An eInsight Business Process can only be a responder, since no eInsight service exists that would allow an eInsight Business Process to invoke JMS Request/Reply functionality. All is not lost, however. Since an eInsight Business Process can invoke a “New Web Service” Java Collaboration as an activity, and a Java Collaboration can invoke JMS Request/Reply functionality, a JCD wrapper can be used to overcome this limitation. The models are illustrated in Figure 4-1.

Figure 4-1. JMS Request/Reply models

The receive method of a Java Collaboration that serves as a responder in the simple Request/Reply model would look similar to that shown in Listing 4-1.

Example 4-1. _Book/MessageExchangePatterns/JMSRequestReply/jcdJMSReqResSrv

public void receive
    (com.stc.connectors.jms.Message input
    ,com.stc.connectors.jms.JMS W_JMSResponse )
        throws Throwable
{
    ;
    // extract request from input JMS message
    String sJMSRequest = input.getTextMessage();
    ;
    // process message content
    String sJMSResponse = sJMSRequest.toUpperCase();
    ;
    // determine whether to send response to a specific destination
    // or to the default Connectivity Map destination
    String sReplyTo = input.getMessageProperties().getReplyTo();
    if (sReplyTo != null && sReplyTo.length() > 0) {
        ;
        // have return address - set destination
        // to return response to where it is expected
        ;
        W_JMSResponse.setDestination( sReplyTo );
    }
    ;
    // send response
    W_JMSResponse.sendText( sJMSResponse );
    ;
}

Needless to say, a responder in a real solution would do something more interesting than convert the request string to uppercase.

Note that this particular collaboration could be used as both a Request/Reply responder and as a regular “pick from one JMS Destination and deliver to another JMS Destination” collaboration.

By obtaining the value of the JMS ReplyTo property in the input message, and setting it as a destination for the response message, we are turning this collaboration into a Request/Response processor. If the component that submitted a message did not set the ReplyTo property, the condition would be false and the response would go to the JMS Destination configured in the connectivity map.

Notice also that this collaboration does not need to be the one that directly interacts with the JMS Destinations set up by the requestor, as shown in the Extended Request/Reply model. There could be other Collaborations and Business Processes operating on the message, with multiple JMS destinations between the requestor and responder. As long as each component in the chain took care to propagate the value of the original ReplyTo property, the response would still get delivered to the original requestor. Propagation of the ReplyTo and other JMS message properties requires a bit more work, as much as two extra lines, as shown in Listing 4-2.

Example 4-2. _Book/MessageExchangePatterns/JMSRequestReply/jcdJMSReqResSrvPreserve

public void receive
    (com.stc.connectors.jms.Message input
    ,com.stc.connectors.jms.JMS W_JMSResponse )
        throws Throwable
{
    ;
    // extract request from input JMS message
    String sJMSRequest = input.getTextMessage();
    ;
    // process message content
    String sJMSResponse = sJMSRequest.toUpperCase();
    ;
    // create a JMS message to send out and populate it
    // with the payload and ReplyTo property value
    ;
    com.stc.connectors.jms.Message jmsResponse
        = W_JMSResponse.createTextMessage();
    jmsResponse.setTextMessage( sJMSResponse );
    jmsResponse.getMessageProperties().setReplyTo
        ( input.getMessageProperties().getReplyTo() );
    ;
    // send response
    W_JMSResponse.send( jmsResponse );
    ;
}

In the previous example, the sendText() method of the JMS Connector object was used to directly send the response string. In order to set properties, a message object is required. The payload and the properties of the object are set, and the message object is sent using the send() method. Needless to say, other properties, including user-defined properties, can be set before sending the message.

The JMS Request/Reply functionality relies on Java CAPS creating a temporary JMS destination, under the hood as it were, and transparently setting the ReplyTo property of the request message to the name of that destination. The requestReply() method puts the request message to a regular JMS destination, named in the connectivity map, and performs a blocking receive on the temporary destination. Once the message is received, or the time expires, the call returns to the collaboration.

The receive method of a basic Java Collaboration that invokes a JMS Request/Reply functionality might look like that shown in Listing 4-3.

Example 4-3. _Book/MessageExchangePatterns/JMSRequestReply/jcdJMSReqRespCli

public void receive
    (com.stc.connector.appconn.file.FileTextMessage input
    ,com.stc.connectors.jms.JMS JMSRRClient
    ,com.stc.connector.appconn.file.FileApplication W_toFile )
        throws Throwable
{
    String sRequestText = input.getText();
    ;
    final int _TIMEOUT_IN_MILLIS_ = 1000 * 30 * 1;
    ;
    // create a JMS Text Message and populate it using input data
    ;
    com.stc.connectors.jms.Message jmsRequest
        = JMSRRClient.createTextMessage();
    jmsRequest.setTextMessage( sRequestText );
    jmsRequest.storeUserProperty
        ( "MyPropertyName", "MyPropertyValue" );
    ;
    // invoke JMS Request/Reply functionality
    ;
    com.stc.connectors.jms.Message jmsResponse
        = JMSRRClient.requestReply
                ( _TIMEOUT_IN_MILLIS_, jmsRequest );
    ;
    // process response, including empty response
    // if request times out
    ;
    if (jmsResponse == null) {
        throw new javax.jms.JMSException
                      ( "Timed out waiting for Response" );
    }
    String sResponse = jmsResponse.getTextMessage();
    ;
    W_toFile.setText( sResponse );
    W_toFile.write();
    ;
}

This Collaboration is triggered by a File eWay and ultimately writes its output to a file using a File eWay. The content of the input file is set as the content of a request message. The JMS Request/Reply method is invoked with a timeout and a request message. The method will return a JMS message with the response or null if timeout occurs. The input message could be delivered by means other than a File eWay, and it could be preprocessed before being sent as a request. The response could be postprocessed and sent to some destination other than a File eWay. The point is that a JMS Request/Reply client is quite simple to implement in a Java Collaboration.

For the simple Request/Reply model, the connectivity map will look like Figure 4-2.

Figure 4-2. __Book/MessageExchangePatterns/JMSRequestReply/jcdJMSReqRespCli

Note the final qDummyResponseNeverUsed JMS Destination. This destination is never used because the Collaboration explicitly sends the response message to the JMS Destination whose name is specified in the JMS ReplyTo property. In this case, it will be the name of the temporary destination created by the JMS Message Server for this requester. Note also that the connectivity map, which could otherwise be used as a good reflection of real connections, no longer accurately depicts the interactions that take place. The literal “Dummy” is added to the name of the unused destination to give a strong hint that code may have to be inspected to discover what is actually happening.

We could have, without loss of functionality, produced a connectivity map where the output of the service svcJMSReqResSrv would be connected to the qSimpleRequestReply. This could, perhaps, be better for the simple case as it would suggest a Request/Response relationship, as shown in Figure 4-3.

Figure 4-3. __Book/MessageExchangePatterns/JMSRequestReply/cmSimpleRequestReply01

For the extended Request/Reply mode, the connectivity map will look like that in Figure 4-4.

Figure 4-4. __Book/MessageExchangePatterns/JMSRequestReply/cmExtendedRequestReply

Here too we could have connected the output of svcJMSReqResSvc service to qPreservePropsQueue to suggest that the Request/Response pattern is used (Figure 4-5).

Figure 4-5. __Book/MessageExchangePatterns/JMSRequestReply/cmExtendedRequestReply01

Note

An important property of the temporary queue, created by the requestReply() method, is that it exists only as long as its creator exists. In this case, if the collaboration that invokes the requestReply() method exits, because the requestReply() timed out, for example, the temporary queue will be destroyed. If the responding service attempts to put a response message to the response queue, it will receive an exception because the temporary response queue no longer exists. This is a very desirable characteristic in Request/Response scenarios where a requestor will not wait longer than a certain amount of time. If appropriately designed, the responder will discard late responses on exception. Note, however, that the responses may be lost by design. Note also that should the system fail while request processing is in progress, message loss will occur.

Much as a Java Collaboration can be used to implement the responder logic, so too an eInsight Business Process can be used for this purpose, as shown in Figure 4-6. Similarly, and even more simply than in the case of a Java Collaboration, the ReplyTo property needs to be copied from the input JMS message to the destination property of the output JMS message. The infrastructure will take care of the rest.

Figure 4-6. __Book/MessageExchangePatterns/JMSRequestReply/bpJMSReqRespSrv

Where building a JMS Request/Reply responder in eInsight is easy, building a JMS Request/Reply requestor is perhaps harder than it needs to be. It is necessary to write a Java Collaboration to wrap the call to the JMS requestResponse() method and invoke that collaboration as an activity in the Business Process. The collaboration will look almost exactly like the one presented earlier except it will be designed as a New Web Service collaboration, and the input and output will be message structures rather than connectors. Chapter 2, section 2.3.1, “JMS Request/Reply Invoker for eInsight,” in Part II, illustrates the steps involved in creating a JMS invoker collaboration that can be used by an eInsight business process to invoke a Request/Reply responder using JMS.

Ultimately, whether the responder is implemented as a Java Collaboration or a Business Process, the JMS-based Request/Response pattern can be used to construct reusable components.

The requestReply() method uses a temporary JMS Destination whose name is transparently set in the request message and is later used by the responder as the destination for responses. The temporary destination, and messages within it, will be destroyed when the Collaboration that created it exits. Since the name of the temporary JMS Destination is transparently generated and set in the ReplyTo property of the outgoing message, (a) it will be different from invocation to invocation, and (b) only the recipient of the message will have access to it. Note also that the requestReply() method variant, used in sample code, specifies a timeout parameter. The value of that parameter must be large enough to guarantee delivery of responses in normal circumstances.

Note

The discussion above uses a somewhat awkward term, JMS Destination, to name what we would naturally call a JMS queue. This is because in the Java Collaboration code samples and Business Process screenshots, there is no distinction between queues and topics, both of which are JMS Destinations. It is not until the connectivity map is being constructed that the actual JMS Destination type is specified. This is handy as the code is generic and destination type–independent and because the same code, if general enough, can be used in solutions using queues and ones using topics.

A form of an Auction pattern, where the fastest responder wins, would be an interesting application of a JMS Request/Reply pattern, which uses topics rather than queues and has multiple responders configured in the connectivity map. Chapter 2, section 2.3.2, “JMS Request/Response Auction Pattern,” in Part II, illustrates this concept with an example.

Note

An important point to note about the JMS requestReply() method is that, at least through version 5.1.3, both the requestor and the responder JMS Clients must be deployed to the same JMS Message Server. If the responder must be deployed to a different JMS Message Server, then other means of implementing a Request/Reply mechanism must be considered instead.

HTTP Request/Reply

The Hypertext Transfer Protocol (HTTP) is the embodiment of the Request/Reply pattern. An HTTP GET or an HTTP POST request is submitted to an HTTP Server, which returns a response. The term HTTP Server is used deliberately to describe a server that implements the HTTP Protocol. To use Web Server would be to invite confusion. Even in the early days of HTTP 0.9, an HTTP Server could return content other than text/html, which is what a “Web page,” as implied in the term Web server, would be. With support for the Multipurpose Internet Mail Extensions (MIME) content-type specification and handling, the range of content types that can be returned by an HTTP Server is virtually limitless. These properties of HTTP are used to implement Request/Response solutions that deal with content other than the Hypertext Markup Language [HTML401]. HTTP makes provisions for PUT, DELETE, TRACE, CONNECT, and OPTIONS methods as well. Some of these, because of the inherent security concerns, are rarely implemented. Others are only implemented to support HTTP Proxies.

Before the advent of Web Services, a number of HTTP-based services were implemented by various parties. As predating Web Services standards, these services typically used HTTP POST to submit solution-specific requests and receive solution-specific HTTP Server responses as content of type text/plain, text/html, or text/xml. The most notable implementation of these is the HTTP Binding for the Electronic Business XML Business to Business Protocol Message Service [ebXMLMS] implementation, which is an HTTP POST Request/Reply–based implementation, and while it uses SOAP over HTTP, it is not a Web Services implementation. These kinds of services may now be rarely developed, but you may find yourself in need of integrating one into the solution you are building. Figure 4-7 depicts the HTTP Request/Response model.

Figure 4-7. HTTP Request/Response model

Java CAPS includes, among others, the HTTP eWay—an Adapter that allows an integration solution to submit an HTTP POST or GET request, or to implement an HTTP Responder that processes HTTP GET or POST requests and returns content of the appropriate type.

Chapter 2, section 2.3.3, “HTTP Request/Response,” in Part II, discusses details of the HTTP protocol, relevant to the examples, and shows a number of example HTTP Requestor Collaborations and an example HTTP Responder Collaboration using both GET and POST HTTP Methods, and returning both traditional HTML content and the XML content.

An HTTP Responder, the Java CAPS component that handles HTTP requests, can be implemented as either a Java Collaboration or an eInsight Business Process. However it is implemented, the responder ought to use information provided in request headers, request parameters, or the request body to perform whatever business function it has been designed to perform. It must then prepare the response body, populate the appropriate response headers, and return the response with the appropriate response status.

Section 4.5.1 discussed the use of JMS in Request/Reply scenarios. The requestor creates a temporary endpoint [EIP] and attaches its address to the message it sent. The downstream processing component, when finished with the message, uses that address as the destination to which to send the response. From the perspective of the requestor, the message exchange is synchronous. Rather than creating a temporary endpoint, a component could listen for messages on a regular endpoint. An upstream endpoint would attach the address to the message, and a downstream component would use that address to send the message to the correct destination. In the former case, the use of a temporary JMS destination is an example of dynamic routing [EIP]. In the latter case, it is an example of fixed routing [EIP] where the upstream component must have the knowledge of the existence of at least one other component in a solution, the listener with a fixed address, and the downstream component must have the knowledge of both the existence of the listener component and the means of extracting the address from the message. This results in tight coupling and interdependence between the three components. This also hides the routing architecture inside component logic and generally results in solutions that are harder to understand and harder to maintain.

Return Address is frequently used in asynchronous Request/Response solutions in which a response may not be ready for some time. It is usual, in these circumstances, to have the requestor submit a request and continue, without waiting for a response, and have a separate listener listen for the response and correlate it with the request.

Asynchronous Business-to-Business (B2B) exchanges are a special case of the Return Address pattern implementation, involving independent cooperating enterprises. The AS2 Protocol [AS2], for example, fairly widely used for secure message exchange in commercial environments, supports the asynchronous Message Disposition Notification (MDN) option—an asynchronous acknowledgment. By agreement between parties, a message sender specifies that asynchronous MDN is expected and provides the recipient with the URL to which to send the MDN. The sending of a message and the receipt of an acknowledgment are decoupled. The sender is responsible for receiving the acknowledgment and associating it with the message to which it applies.

Correlation, the ability to collect related messages and process them together, is a sufficiently interesting matter to warrant an independent chapter. Chapter 11, “Message Correlation,” contains a comprehensive discussion of Java CAPS correlation facilities using both the eInsight Business Process Manager Correlations and plain eGate-based facilities.

[EIP] discusses requirements for message sequencing in the context of support for transfer of very large messages as a sequence of message chunks. Messages are broken up at the source, or at the entry into the messaging solution, and reassembled into a single message, or delivered to the ultimate destination, in order. It discusses the subject with a view of suggesting the means of attaching sequence numbers and message counts to messages as an aid to reassembling messages.

There are other circumstances when ensuring that message sequencing is preserved is critical to correct operation of the messaging solution. In healthcare environments, for example, where messaging solutions may be transferring laboratory results for patients, it is critical that a message carrying preliminary results or carrying partial results, typically sent first, arrives at the ultimate destination before a subsequent message that carries final results or updates, or supplements results carried by the original message. People’s lives may depend on the preservation of message order.

In the case of fragmentation and reassembly of messages, as discussed extensively in [EIP], the Splitter is the component that breaks up large messages into clusters of smaller, related messages. It enriches each message fragment with the unique sequential identifier that identifies the cluster of related messages. It also enriches each message with the message sequence number identifying the position of the message within the cluster of messages and the “size” of the cluster. The Aggregator is the component that reassembles related fragments into a single message or that is responsible for submitting messages in order to some endpoint. It uses the cluster ID, the message sequence number, and the cluster size to collect related messages. It establishes the order of messages and detects when all the messages in the cluster have been received. Components that support this kind of functionality would implement some combination of the Correlation pattern, the Splitter pattern, the Aggregator pattern, and the Envelope Wrapper pattern. Correlation functionality would be required to allow the component to collect related messages. A Splitter might be required to break up a large message into smaller chunks. An Envelope Wrapper would be used to enrich each message with the metadata necessary for reassembly. Aggregator would be required to combine all related messages.

Note

As a general proposition, transferring very large messages through messaging systems is not a good idea. In many cases the messaging system must use an amount of memory many times the size of the largest message it is to process. This could lead to a memory shortage issue that could become particularly severe when dealing with large XML messages or contents of entire multimegabyte files. In general, the presence of very large messages in a messaging solution is an indication of poor solution architecture. The presence of large messages will always degrade performance and will likely lead to scalability and resilience issues. Where large XML documents may not be able to be broken up, content of large non-XML files may well be able to be processed using the Batch eWay in a streaming mode combined with the Batch Record eWay breaking data up into “records” and passing these on as soon as available. Depending on the relationship between records, implementation of a Message Sequence pattern may well be required to allow reassembly. The use of Batch eWay streaming is discussed in section 4.11.1.

Explicit message sequencing would use an Envelope Wrapper to carry sequencing metadata. It would use a Splitter and an Aggregator to break up and reassemble large messages, or ensure sequential processing of message fragments. Implementation of Aggregators in particular is reasonably complex. It requires the use of some form of Message Correlation and message store, in addition to the logic required to determine sequence commencement and completion. While such components could be made reasonably generic, or built as templates for reuse, they require a nontrivial development effort.

Java CAPS offers facilities that can be used to guarantee message sequence without explicit implementation of Message Sequencing components.

The Sun SeeBeyond JMS Message Server can be globally configured to support different FIFO processing modes. One of the modes is a fully serial first in, first out (FIFO) mode where receivers can only read a message after all previously read messages have been received and committed [JMSRef]. Serialization of message processing using JMS Message Server FIFO modes affects the behavior of Competing Consumers. This is discussed in the following sections.

Sun SeeBeyond JMS Message Server FIFO Modes

The following discussion applies only to the Sun SeeBeyond JMS Message Server implementation.

In an ordinary course of events, when there is only a single receiver receiving from a JMS queue and it is configured to use the Serial Concurrency mode, messages will be received in the order in which they were queued. Two or more receivers configured to receive messages from the same JMS queue will compete for messages. If messages are arriving faster then a single receiver can process them, then when one receiver is busy another receiver will get a message to process. The receivers/ consumers compete for messages. In the presence of Competing Consumers, processing behavior will vary with the JMS Message Server’s global FIFO Delivery mode configuration. Each of the FIFO modes—Fully Concurrent Processing, Protected Concurrent, and Fully Serialized—has a different effect on concurrency of Competing Consumers and the ultimate order of messages processed by them.

Chapter 2, section 2.4.2, “Sun SeeBeyond JMS FIFO Modes,” in Part II, illustrates behavior of Competing Consumers with different FIFO modes with a detailed example.

With the default global setting of Fully Concurrent Processing FIFO Delivery mode, all receivers will run in parallel and compete for messages. This will result in messages being processed in random order.

The Protected Concurrent FIFO mode ensures that order is preserved but allows parallel processing to still take place. This mode can be considered when designing solutions that can scale across multiple processors, for example.

The Fully Serialized FIFO mode causes message processing to be fully serialized. Message order is preserved, but there is no more parallelism in the solution.

Note

It is important to remember that the queue consumer concurrency setting is not graphically represented in the connectivity map, so a casual peruser may miss this important configuration setting. Also, it is important to remember that the special FIFO Delivery modes are not accessible through the connectivity map at all, but must be looked for through the Integration Server Administration console.

Note that message sequencing functionality provided by the JMS Concurrency and FIFO Delivery modes applies to all messages regardless of their relationships, if any. It could be that messages are somehow related. For example, a Patient Admission message, a Laboratory Request message, a Laboratory Results message, and a Patient Discharge message, all belonging to the same patient, might be expected to be processed in the order they were enumerated here. To process a Patient Discharge before Patient Admission, or Laboratory Results before Laboratory Request, may be incorrect, but processing a Patient Discharge message for one patient before a Patient Admission message for a totally unrelated patient may be perfectly valid. Support for this kind of message sequences requires the solution to explicitly recognize message relationships and to provide explicit message sequence–processing facilities. This issue was discussed in the opening part of this section.

Serializing Business Processes via JMS and XA

Ordinarily, the eInsight Business Process Engine will receive a JMS message and acknowledge its receipt to JMS as soon as it manages to create an instance of the process and deliver the message to its Receive activity. This is in contrast with the behavior of a Java Collaboration where the JMS message is not ordinarily acknowledged until the collaboration completes processing and returns. This implies that, in ordinary circumstances with many messages in a queue that feeds the process, there may be many Business Process instances concurrently processing messages. In effect, this is the same behavior as that exhibited by a Java Collaboration using the Connection Consumer Mode Concurrency and an unlimited pool. In the case of an eInsight Business Process, however, the setting of the Concurrency mode on the JMS receiver client makes no real difference. Since many eInsight Business Processes execute concurrently, it is very unlikely that messages will be processed in the order in which they were enqueued.

Java CAPS 5.1 extended the traditional Business Process Execution Language (BPEL) model to provide the capability to impose distributed transaction semantics on an eInsight Business Process. With no XA transaction, an eInsight Business Process would acknowledge the receipt of a JMS message as soon as it received the message. The process could then take any amount of time to process the message to completion. A new instance of a business process would be created as soon as another message became available in the JMS Destination from which the process was receiving messages. Both process instances could conceivably be executing concurrently, and the “younger” process could conceivably complete processing its message before the “older” process processed its message. This would cause the subsequent component to receive messages out of sequence, possibly necessitating resequencing. With a Serial Consumer mode and XA transaction semantics defined for the Business Process, a new instance will not be created until the first instance completes processing its message and commits. This will again guarantee that messages are delivered to the downstream component in the order in which they were received.

Chapter 2, section 2.4.3, “Serializing Business Processes with XA,” in Part II, provides detailed examples illustrating this discussion. Business Process behavior both with and without XA is illustrated.

Note

Note that imposing XA semantics on eInsight Business Processes is not to be done lightly. Business Processes are intended to be long running, on the order of minutes, hours, days, or longer. Imposing XA semantics on long-running processes may lock resources that should not be locked for extended periods of time, databases for example. Keeping locks on such resources may result in increased resource contention, resource starvation, or deadlocks. This will affect both the stability and the performance of a solution. One should carefully consider implications of enabling XA on a process-by-process basis.

While Java CAPS provides facilities that can be used to ensure processing of messages in order, there will still be circumstances when these facilities will not provide the desired result. Ensuring break-up of messages and reassembly of fragments must be handled with solutions that implement Splitter, Aggregator, Envelope Wrapper, and Correlation patterns. Ensuring sequential processing of related messages requires similar solutions. Specific implementations of the Splitter, Aggregator, Envelope Wrapper, and Correlation patterns are discussed elsewhere in the book.

The Sun SeeBeyond JMS Message Server supports message expiration. Expiration can be configured globally through the Integration Server Administration Console interface, as shown in Figure 4-8, and specified individually for each message by assigning appropriate values to the timeToLive property, as shown in Figure 4-9.

Figure 4-8. Assigning timeToLive value

Figure 4-9. Assigning timeToLive value programmatically

By default, Maximum Lifetime is 30 days.

When message expiration is disabled, or the Maximum Lifetime is set to 0, messages that are not delivered to their destinations will be retained indefinitely in queues and in topics with durable subscribers.

Java Collaborations with outbound JMS connectors can be configured to assign the number of milliseconds, as a long integer, to the timeToLive property; see Figure 4-10. This allows the designer to explicitly specify message expiration. A variant with the explicit time-to-live parameter can also be used to specify explicit expiration when using one of the explicit JMS OTD send methods.

Figure 4-10. Assigning timeToLive in a collaboration

The message expiration property in the incoming JMS message specifies the Greenwich mean time (GMT) at which the message would have or has expired. While a message whose expiration time is in the past is supposed to be discarded by the JMS Message Server implementation, the JMS 1.1 Specification [JMSSpec] suggests that this may not always be the case. The Specification states that:

Chapter 2, Section 2.5, “Message Expiration,” in Part II, illustrates message expiration with Java Collaboration examples.

If the receiving component does not set the value of timeToLive on the message it sends to the JMS queue, the JMS Message Server global Maximum Lifetime property will apply. By default, the message will sit in the queue for 30 days or until received by some component, whichever is sooner.

We can determine how long the message took to deliver from the sender to the receiver, if such knowledge is of interest, by setting a known timeToLive value in the sender and examining the expiration value in a receiver immediately downstream.

If Message Journaling was enabled, the message would have been journaled. Journaling takes place upon message delivery or message expiry.

Like Java Collaborations, eInsight Business Processes also have access to JMS timeToLive and expiration properties, as well as the current time in milliseconds; see Figure 4-11.

Figure 4-11. TimeToLive and Expiration properties in a Business Process

By using GMT, also known to the politically correct as the UTC (Universal Time Coordinated), we avoid issues that arise when messages cross time zones or when summer time/winter time crossovers occur while the message is in transit.

Solution designers can use the mechanism, in which a time offset or time-to-live is added to the time message that was created or sent, in GMT, to explicitly handle message expiration regardless of whether JMS is used. If expiration is needed, one or more Envelope Wrapper solutions can be used to carry the expiration time. Components can determine whether to discard or process messages after comparing expiry time to current time. In fact, using an explicit expiration time metadata item simplifies the solution by allowing any component downstream from the expiration setter to determine if the message expired, rather than having to reset the timeToLive parameter on each send, as is required when using JMS.

The Sun SeeBeyond JMS Message Server provides the Message Journaling service to allow copies of consumed or expired messages to be saved for inspection or reply at a later time. Journaling consumes resources; therefore, by default, Java CAPS JMS Message Server does not enable message Journaling. When enabled, through the Integration Server Administration Console interface, Journaling will apply to all messages carried through the JMS infrastructure. An Expiration Interval can be globally associated with messages held in the Journal. Unlike expired live messages, the expired journaled messages are not removed from the journal. Solution designers who use JMS Journaling must ensure that operational procedures are put in place to purge expired messages from the journal using command-line tools provided with the product.

Message Journaling is discussed at some length in Chapter 5, “Messaging Infrastructure,” section 5.6.11.

It is possible, even likely, that as a messaging solution evolves over time, message structures it uses will change. The change may be a disruptive one, where the new message format is completely different from the old message format originating in an external system, because the external system was upgraded or replaced. The old system will be retired, and the new system’s messages will be the only ones the infrastructure will see at runtime. The change may be evolutionary, where new data fields are added or existing data fields change size or data type. Data in both the old format and the new format may be present at runtime concurrently and must be correctly processed.

How a Java CAPS solution can address these kinds of issues depends on message sources and the designer’s ability to anticipate changes.

An eWay Endpoint may be format-specific to the point that it must be reconfigured or replaced to accommodate the change. A HL7 eWay, for example, can be configured to look for and validate certain fields in incoming HL7 messages. HL7 eWay’s ability to validate data requires configuration of the HL7 version in use and sending and receiving application identifiers. If an external application starts sending HL7 messages of a different version, validation at the eWay may fail. To resolve this issue, the eWay would have to be reconfigured, and the solution that contains it would have to be rebuilt and redeployed. If it is expected that different HL7 versions will be processed by this eWay, validation may be disabled. Disabling validation at the eWay defers dealing with the issue of multiple versions to a component further downstream.

An endpoint may be message-format agnostic. A Batch Local File or a Batch FTP eWay, for example, deals with files. Neither deals with the structure of the files. Changing the format of messages in files will be transparent to both.

A JMS Client is message-format agnostic to a large degree. It differentiates between text messages, bytes messages, stream messages, and map messages, but within each it does not care whether the intrinsic format of the message, represented as a text message, for example, changed or not.

In most cases it is a Java Collaboration or an eInsight Business Process that has to account for changes in message format. Since in most cases messages are received as blobs, and are unmarshaled into OTDs for manipulation, the unmarshal step will fail if the message format does not agree with the format the OTD expects. Depending on the source of messages, and the way the solution designer deals with adversity, an unmarshal exception may result in message loss. When the message source is a JMS Client and the component that deals with the message is a Java Collaboration, for example, an unhandled unmarshal exception will cause the message to be rolled back to the JMS Destination from which it came. Since the same message will be resubmitted some time later, the cycle of unmarshal–rollback–resubmit will be entered and will only end when the message is removed, either manually or by a redelivery handling mechanism if configured. A designer could explicitly discard a message that fails to unmarshal by “handling” the exception in the Collaboration or the Business Process, causing it to return normally (i.e., consume the message). Neither of these solutions handles unexpected messages gracefully.

Since JMS messages contain both the payload and a series of properties, one of the solutions to the problem of recognizing different message formats would be to introduce a Format Indicator, a JMS user-defined property that carries an indication of what kind of message is being carried. This Format Indicator could allow Java Collaborations and Business Processes to choose, at runtime, which OTD to use to unmarshal the message. The Collaboration or the Business Process could be constructed to handle messages differently depending on the format.

Some component would need to determine what the format is in order to set the Format Indicator. One way to handle this would be to allow a component, early in the solution, to try to unmarshal the message to different OTDs and, depending on which succeeded, set the Format Indicator. Another, which is dependent on the nature of the message, would be to inspect some leading part of the message blob to attempt to deduce what kind of message is being carried.

It is possible that a message may need to traverse a Messaging Bridge, thus losing the metadata that could be carried in JMS user-defined properties. If the Message Format Indicator was carried as a JMS property, it too would be lost.

It would be best to design messages such that messages themselves carry a Format Indicator and are designed using techniques similar to those discussed in Chapter 7, “Message Construction,” section 7.4, so that only a partial unmarshal is necessary to determine message format.

Format Indicator could also be used to handle multiple versions of a message interface. The indicator would be set to indicate to which version of the messaging interface the message conforms and would then be used by a content-based router to send the message to the component that implements the correct version of the interface. This would enable new versions to run alongside the old version while still using the same message infrastructure.

As mentioned in section 4.8, handling of very large messages in a messaging solution may require memory resources many times greater than the size of the largest message to be handled. Frequently, the architect has no choice but to consume or produce a very large message, a file containing a batch or related transactions, for example, or a large and complex XML message generated by, or intended for, an external application. Handling such messages poses special challenges. Java CAPS can assist with Batch eWay support for data streaming when such messages are manifested as files in a file system. It is possible to break up large messages into components and process components individually, or collect components and assemble them into a large message. eTL, another of the products in the Java CAPS Suite, can assist in processing large volumes of data. While ETL (Extract, Transfer, and Load) is typically associated with one off-batch extraction and load of data, Java CAPS’s eTL can be used both standalone and in-stream as part of a larger Java CAPS solution. We discuss eTL in this in-stream mode as a possible means of streaming data between a flat file and database table or between database tables and views.

Batch eWay Streaming

Java Collaborations that use Batch eWays are commonly implemented to read the entire content of the file into the Payload node of the Batch eWay OTD, then unmarshal it into some OTD that gives access to individual “records” or fields. If the file contains multiple records that will be processed individually downstream from the Batch eWay, it would be far more efficient to read the file a record at a time and release records for further processing as soon as available. Batch Record eWay provides the ability to stream file data into a parser that breaks it into records. The parser works with delimited, fixed-length, and “whole file is a single record” records through connectivity map–based configuration properties.

Batch eWay Streaming Adapter can be used for streaming data between an FTP Server and a local file, in either direction, without the need to use the Batch Record eWay. This is useful if a Java CAPS solution needs to obtain a large file from an FTP site for use locally or needs to send a local file to the remote FTP site as efficiently as possible. Chapter 9, “Messaging Endpoints,” section 9.4.2, uses an example of local file-to-FTP Server streaming to illustrate polling JMS Destination as part of exception and retry handling.

Chapter 2, Section 2.6.1, “Batch Local File Streaming,” in Part II, illustrates the concept of data streaming with several detailed examples, including the use of Batch Record, Java Buffered IO in conjunction with BatchLocalFile eWay’s Stream-Adapters, and eTL.

To stream a local file in, process its contents a piece at a time, and stream it back out again, you must use a BatchLocalFile and a BatchRecord for the inbound side as well as a BatchRecord and a BatchLocalFile for the outbound side. Because files are streamed, this collaboration can process files of arbitrarily large size without grossly inflating JVM memory use.

Note

For fixed record files, all records, including the last record, must be the same size. This means that you cannot use the batch record with an arbitrary-size record to emulate buffering unless it so happens that the last record is the same size as all other records. When it is not, the last read will fail and the bytes remaining to be read will not be accessible and will be lost. Exception will be thrown when that happens.

Streaming can be used to process files containing fixed-length records. Rather than specifying Record Type as Delimited, we specify it as Fixed and furnish the appropriate value for the Record Size.

Note

Specifying Record Type of Single Record is no different from not using the BatchRecord at all but instead using Batch eWay, in one of its variants, to load the entire file into memory as a single record.

If it is absolutely necessary to break a file into “buffer-sized” chunks without regard for delimiters or the size of the file, where the file is not guaranteed to have an even length that is a multiple of desired buffer size, it is still possible to stream the file. You would have to use a BatchRecord eWay with a fixed-length record of 1 byte on the input side and do your own buffering.

With Batch eWay, there are good, not so good, and downright bad ways of processing large volumes of data, as demonstrated in the examples. Which of the potential solutions for data streaming is best will very likely depend on the individual requirements.

In some circumstances it may be necessary to ensure confidentiality and integrity of messages that travel through the messaging system. One or both of two methods are typically used to protect messages in transit.

More common and easier to relate to, because of its ubiquity, is the method that secures the point-to-point channel over which messages travel. Channel Security is typically provided by adding the Secure Sockets Layer (SSL) to the TCP/IP Protocol-based channels. SSL standard [SSL] specifies how encryption can be applied to all bytes traveling through a point-to-point channel. It also specifies how endpoints can authenticate each other, exchange cryptographic material, choose encryption algorithms, and negotiate protocol version. In essence, and this is trivializing the matter considerably, to complete establishment of a Secure Session, the two endpoints perform a Cryptographic Handshake, during which capabilities, cryptographic material, and credentials are negotiated and exchanged before any payload data is sent. If the SSL Handshake fails, no session is established and the endpoints do not exchange data. If the SSL Handshake succeeds, the endpoints cooperate, for the reminder of the session, in encrypting on send, and decrypting on receive, the byte stream that represents the payload data. Java CAPS provides the SSL-based Channel Security capability in the HTTP Client and the HTTP Server eWay, in the JMS Message Server implementation, and in the Web Services implementation framework. See Chapter 14, “Security Features,” section 14.4, for a discussion on SSL configuration in Java CAPS for solutions using HTTP eWay and Web Services endpoints.

Less common is the method that individually secures each message. Since encryption and digital signature attributes are applied to the message itself, a secured message can traverse multiple channels and multiple components while preserving security. The Secure Messaging Extension eWay, available in ICAN 5.0.5, reappeared late in 2007 in conjunction with the release of the Sun B2B Suite 5.1 and the AS2 Protocol Manager, a successor to the eXchange Integrator of the 4.x and 5.0. It supports the Secure Multipurpose Mail Extensions (S/MIME) for encryption and digital signing, the XML Digital Signatures, and the XML Encryption functionality. The SME eWay provides an easy means of securing individual messages.

Chapter 14, section 14.5, discusses how data can be secured in transit using secure variants of the Batch eWay. These variants operate on files using different transfer security mechanisms.

This chapter discussed Java CAPS facilities and techniques used to implement specific message-related patterns that [EIP] groups together under the heading of Message Construction. The following patterns were discussed:

The chapter discussed a number of concepts and tools. In addition to regular Java Collaborations and eInsight Business Processes, the eTL tool was introduced, and its use in data streaming applications was discussed.