According to rock-and-roll legend, Johnny Rotten once said, “I don’t know what I want but I know how to get it.” He didn’t realize it, but he was predicting the current state of event-driven architecture (EDA). The uses of EDA are many, and the range of misunderstanding about them is just as impressive. Before we plunge into the practical issues involved in implementing EDA, we thought it would make sense to catch our breath for a moment and look in more depth at why EDA is a good idea for some (but not all!) enterprise computing scenarios.
EDA has the potential to transform IT and computing on many levels. The paradigm can improve existing modes of IT, as well as introduce wholly new ways of doing things. Although there are many places where these improvements can be realized, we would like to focus on four primary areas. EDA is poised to make sweeping changes in enterprise computing, particularly in cost reduction through outsourcing and better system management. EDA has the potential to render great advances in agility and analytics. EDA will also be a boon to computing in the general societal level, including government and academia. Finally, EDA has the potential to improve compliance, a high-pressure area for many in the IT field.
Many discussions of EDA focus on how the paradigm can benefit businesses or organizations through improved analytics or responsiveness to events. However, EDA also has great potential to improve the way IT itself is conducted, so we want to highlight several areas where EDA can benefit the IT organization on its own turf. The opportunities for EDA in IT systems management fall into three basic categories, which we call “old dog, new tricks,” “new dog, new tricks,” and “new dog, no tricks.” The last category covers innovations from EDA that don’t quite exist, although some people think they do.
System monitoring is an existing area of IT that EDA has the potential to transform. As IT infrastructure grows in both scale and complexity, EDA can help system monitoring software work smarter. For example, if we consider data about server load levels to be events, we can set up event listeners to track server load levels. An EDA-based system monitor can intelligently process the server load event data and respond accordingly. To some extent, system monitoring software already works this way, but the standards-based pub/sub design, and the inherently dynamic characters of an EDA, can enable a higher level of system monitoring responsiveness to changing server usage patterns. The interesting point here is that the system monitor can learn from the causality of problems and then send future alerts warning of potential problems before they happen. If fact, many of the infrastructure monitoring suites are beginning to add this functionality now.
Security monitoring is another area where EDA can bring about positive change. A relative of system monitoring, security monitoring is a collection of activities that involve the generation, audit, and interpretation of any number of security-related system logs. For example, a large enterprise might simultaneously generate logs of system use, logins, firewall access attempts, intrusion detection warnings, and so on. Figure 4.1 shows a network that has multiple security logs for its routers, firewalls, servers, and so on. Analyzing these logs can be difficult, or even impossible. Certainly no human being can look over a thousand log records an hour and put them into any kind of order. Storing and outsourcing of log interpretation is a costly necessity for many organizations. Interpreting security events across multiple logs can be highly challenging, and that is where EDA can come to the rescue.
A security monitoring EDA can establish event generators to analyze the event stream of log entries, looking for patterns, and producing events based on analytical criteria. The event processors can then correlate actions across security logs and produce security alerts in real time for consumption by system administrators and security personnel. An EDA-based security monitoring application is represented in Figure 4.2. By analyzing the security logs in parallel, and in real time, the EDA security monitor can catch patterns of events that might elude the individual analysis of security logs. With today’s hacker attacks growing more sophisticated, this kind of real-time multilevel security analysis can make the difference in detecting an attack before it begins.
Of course, you could set up multiple streams of monitoring through conventional application integration techniques, and to some extent, security monitoring already works this way. However, the breakthrough of an EDA approach is in the dynamic processing and loose binding used. If connecting a security log to a security monitoring and analytics system requires too much time and money, it either won’t get done at all or the integration project will be scheduled for when time permits. If connecting, disconnecting, or reconfiguring a security log’s integration with a security monitoring system is relatively simple, fast, and inexpensive, as it could be with an EDA, then that security monitoring system will have a lot more ability to be on top of security data in real time.
The flexibility of the EDA approach also allows for new monitoring to be added quicker. As new hack approaches emerge, it’s much easier to add additional event processors to monitor and mitigate the risk. These new forms of attack could even come from external sources, meaning companies could share the attack approach upon initial detection, hence thwarting subsequent attacks on other companies. This demonstrates the wide awareness concept of EDAs.
The other existing area of IT where EDA can bring change, mostly in the form of savings in time and money, is software integration. Most modern commercial software packages offer application programming interfaces (APIs) to enable integration with other applications. However, integrating through proprietary APIs can require extensive reengineering. Service-oriented architecture (SOA) and Web services can both improve application integration speed and budget through the use of standards. However, EDA can do even better because of the way that EDA revamps the whole approach to application logic. Let’s assume that the conventional, proprietary middleware approach to enterprise application integration (EAI) is the slowest and most costly. SOA/Web services introduce a degree of speed and cost improvement, but in most cases the application logic would remain the same. That is, the applications being integrated would still process data on their own and share it with other applications through the Web services API. In contrast, an EDA approach would pull apart the tiers of application logic and move certain logical operations into distinct event processors loosely coupled together to provide the desired result.
As an example, look at Figure 4.3, which depicts a simple application integration scenario. In the phone company case, the line management system needs to update the billing system regarding which accounts have an overage in their minutes. In the baseline case, shown at the top, the company sets up proprietary middleware EAI, which sends a MinuteOverage message from the line management to the billing system, and then ferries back a confirmation message. As we know, although there is nothing wrong with this approach, it can be costly and slow to change. Modifications to either system might necessitate changes to the middleware, and so on.
In Scenario A, the company uses a basic Simple Object Access Protocol (SOAP) Web service to achieve the same application integration. In Scenario A, the line management system exposes a Web service that can send MinuteOverage messages upon invocation by its own system rules. The MinuteOverage SOAP message triggers a SOAP response of confirmation from the billing system that travels back to the line management system. This approach saves the company the expense and complexity required to maintain the proprietary middleware. However, it is still a point-to-point solution, and might in fact be fairly fragile if the systems are changed. Overall, it is still simpler and less expensive to maintain and change than the proprietary middleware, but there is a better way.
Scenario B, shown at the bottom of Figure 4.3, illustrates an EDA approach to this kind of application integration. Though EDA is not an application integration technology, per se, EDA can result in a very agile integration approach. Imagine that there are two event streams at work here: a series of MinuteOverages and a series of UnPaidAccount updates, which flow into a message hub. Event listeners can review the event streams and pick out correlated pairs of MinuteOverages and UnPaidAccounts. The event processor can then respond to customer status requests from the order management system.
This approach, while perhaps time consuming to implement and costly to initially set up, enables very fast changes in application integration moving forward. With an event bus as the enterprise’s core of application integration, and each application set up to listen for and process event data using standards, application integration becomes a matter of plugging and unplugging to the event bus and modifying application logic in discrete locations. If the EDA has been established with a low level of maintainability and preconception among the components, this is possible.
Overall, EDA has the potential to improve the speed and cost characteristics of a number of existing IT activities. Although these activities, such as system monitoring, security monitoring, or application integration, are not new at all, the EDA can make them different and expand the range of possibilities that can be achieved in IT through these existing activities.
EDA offers several new functions to enterprise IT. Or, at the very least, these functions are so highly innovative as to be considered new versions of existing practices and functions. These include new application models, agility of architecture, and adaptability and reuse of system components.
Reuse of IT assets is a potentially beneficial quality of EDA and SOA. By breaking applications into universally accessible components, or services, the event exposure of EDA enables reuse of those services across multiple composite applications. For example, an event listener that detects stock market fluctuations could be used by a trading application, a stock market research application, a mobile market reporting application, and so on, without any modification to its basic design or communication interfaces. In the EDA context, the potential for reuse spans beyond that of Web services in the context of an EDA. They are so isolated from one another, in terms of preconception and maintainability, that they can be reused with a high degree of efficiency. Just to be clear, we are not saying that reuse is always justified or even desirable. There are certain situations where software reuse is a great way to be agile and economical, but others where it is not worth the extra effort involved. And, governing reuse and managing load levels can be challenging, but that is a topic we take up later on in this book.
The value of code reuse emerges in composite applications. EDA, like SOA, enables the development of composite applications, software that functions using a collection of software modules, most likely SOAP using Web services, that function in concert to accomplish the requirements of the program.
Though not exactly new, the standards-based approach of SOA and EDA to composite application design changes the way that IT manages software assets. In contrast to conventional software deployment, where an application is typically installed on a server, or group of servers as a stand-alone functional entity, the composite application resides on many servers, including ones that might be outside of the enterprise itself. There are many benefits to this approach in terms of cost flexibility and management. At the same time, the increase in flexibility can increase complexity and must be managed accordingly.
Figure 4.4 shows an example of event services being reused in more than one composite application. In this case, we have a consumer portal that offers an auto loan referral service, which is a lead generator for the auto loan finance companies, as well as banner ads from car companies. The consumer portal uses an EDA-based application to correlate the event streams of loan referrals and car ad clicks to generate lists of highly qualified leads for specific cars. If a site visitor requests information on an auto loan and also clicks on an ad for Ford Mustang, for example, then the EDA application identifies that person as strongly interested in a Ford Mustang. The potential for reuse and composite application development comes from the many add-on applications that can be created out of these existing modules. The event streams coming from these two modules—the auto loan referral and ad server—can be reused to create consumer analytics applications, credit repair referral services, and follow-on ad server functions.
Overall, though, the EDA and SOA approaches to application design result in an increased agility for enterprise architecture. As it becomes simpler to unplug, move, and reconnect EDA components dynamically with the help of standards, the IT change process will begin to speed up. EDA also enables enterprise architecture to be more adaptable to changes in its required use than conventional architecture.
Like all new paradigms, EDA has generated some dubious potential capabilities. Whether they are the product of misunderstandings by early adopters or vendor excitement, these faux innovations of EDA can be confusing to the rest of us. We want to focus on a couple of IT functions for which EDA does not provide much in the way of innovation.
Service virtualization, for example, is an IT trend that is largely separate from EDA. A virtual server replicates the functions of one server on any number of other machines, a process that increases the efficiency of data-center asset utilization. Although EDA system types can help manage virtualization by monitoring virtual machine usage and activity levels, the EDA itself does not create the virtual machine.
Similarly, grid computing (sometimes known as utility computing) is an IT model that can use EDA for management and systemic self-awareness, but the EDA is not essential to its design.
We’ve already said that EDA leads to greater agility in business, but now that we know more about the workings of EDA, we can take a much more in-depth look at why this is the case. In the Preface, we described agility as a process that involves a cycle of activities, shown in Figure 4.5. To be agile, an organization must be able to perceive circumstances in a way that indicates that an agile move might be required. Based on the information received, and the logic used to assess this information, the organization might elect to make a move, or not. Either decision—yes or no—could be the hallmark of an agile organization. The better the assessment logic, the better the agility will be. If the organization makes a move, it must execute on the decision, a process that typically involves information systems on multiple levels. Then, the organization must assess whether its agile move was a good idea or not, and repeat the process once again, with fresh, postexecution data feeding into the “perception” step of the agility cycle.
Strategic agility refers to an organization’s capacity to change its strategic plan or business model in a rapid enough cycle to match changing business conditions. Operational agility, the ability of an organization to modify how it operates in response to changing conditions, is the flip side of strategic agility. Typically, a strategic shift necessitates a reworking of operations, with a concomitant shift in IT. Functional agility is a measure of how well a company can change its ongoing business processes in response to business conditions.
EDA has the potential to improve all three types of agility. As the best possible “nervous system” for the enterprise, EDA is ideally suited to serve the needs of perception, reaction, execution, and assessment that form the core of agility. Best of all, the “react and plan” function is really reacting and planning, not simply following a sequence tree of predefined logical condition matching. In strategic terms, an EDA can provide high-quality data about business conditions that relate to strategic decision making. EDAs have the capacity to deliver real-time (or near real-time) data about business conditions. Although some strategic decisions need to be made too slowly to benefit from real-time data, most strategic thinking is influenced by the latest information about a particular business situation. Certainly, the more accurate and timely data that strategic decision makers have, the better off they will be. EDAs also give strategists the ability to correlate data from diverse sources and observe hidden trends over long periods of time. This is an advantage that EDA delivers to strategic decision making.
Operationally, the EDA’s flexibility and IT asset reuse enables the kind of fast operational changes that a strategic agile move might require. In particular, a well-designed EDA can speed the kinds of business transitions that occur in mergers or reengineering projects that are so often associated with agile decisions. The loosely coupled nature of EDA components makes them relatively quick and inexpensive to move around and reconfigure, so business transformation with EDA can be faster than those done using conventional middleware approaches to EAI. There are a number of challenges to this simple idea, and we explore them in depth as we continue in the book.
In the functional area, the EDA’s flexibility and ability to sense real-time or nearly real-time changes in business conditions can result in a high degree of agility. EDA can be a key enabler of what the industry calls real-time analytics, systems that feed information to decision makers rapidly through management dashboards and reports. The event listeners and processors can look for specific patterns of events and report them to decision makers. Moreover, the monitoring of events lends itself to learning, through event pattern recognition, allowing business alerts to be issued even before human detection.
EDA also has great potential to enable the use of business process modeling tools, or BPM. BPM is a set of technologies and business practices that breaks down a business process, such as sending out an invoice, into a discrete process step, for example, “Look up customer address” and “Seal envelope” and correlates it to underlying IT systems where appropriate. BPM is useful for mapping business management’s requirements to IT capabilities and systems. And, while BPM can be used with many different enterprise architectural paradigms, the loosely coupled quality of EDA has the potential to render BPM the most effective and agile.
Looking forward, the EDA will actually make BPM tools of today obsolete. In EDA, it’s the events that drive the flow and transition of state, not a BPM controller. In an EDA, the BPM tool simply provides insight to the listeners, how they’re linked, and who is processing what. The linkage representation (matching events to listeners) is an important part because this is all done in real time. The EDA BPM becomes a looking glass into the processing of information instead of the traditional controller model.
However, when we talk about EDA being the paradigm that has the potential for the greatest levels of agility, we are assuming here that the EDAs have been designed adequately for the tasks of perceiving changes in the business environment, and that they are sufficiently standards-based and decoupled to ensure the kind of flexibility that is needed for all types of business agility. There is nothing inherently agile about an EDA, especially if it has not been designed to help assist management in making agile decisions.
The power of EDA reaches far beyond the boundaries of corporate IT. Computers and software play a major role in virtually every aspect of society and public life, from health care to government, education at all levels, and the sciences. EDA, with its real-time responsiveness to changes in conditions and its rapid change cycle, has the potential to alter, and in some cases improve, the way that society uses computing.
The old saying, “Figures don’t lie but liars always figure...” is often used to refer to politicians and the policy “wonks” who feed them information about the state of the world and what ought be done about it. Where does all this information come from? Much of it comes from databases, public and private, that store information about diverse but related areas of interest to policymakers. For example, a policymaker might say that obesity in America is connected to the rise in the use of corn syrup in processed foods. Whether this is actually true or not is open to debate, and it not our intention to get in the middle of any contentious issues. However, what we want to point out is that a public policy discussion that links obesity with corn syrup consumption is one that connects two wholly separate sets of data and places them into the same argument.
There is data about obesity and there is data about corn syrup. There is a correlation between the two trends: Obesity is on the rise, and so is corn syrup consumption. However, that does not necessarily mean that there is a cause-and-effect relationship between the two trends. Perhaps there are other factors that influence obesity as well, such as excessive television viewing, lack of exercise, obesity-producing hormone additives in meat, and so on. How do those factors contribute to obesity? Or, perhaps, one or more do not figure in? An EDA approach to researching obesity could give policymakers more accurate data about the true nature of a trend and what needs to be done about it.
Now, you might think, “Do you need an EDA to compare trends in different data sets? Can’t you do that with Microsoft Excel or any number of other techniques?” Yes, of course you can, but an EDA gives you several capabilities that can be quite tricky to pull off using standard techniques. For example, you can observe trends in multiple data sets in real time, and continually add data sets for comparison or modify the way they interact in real time. EDA gives you a high level of control and flexibility over dynamic data analysis.
Taking a look at a more urgent and practical example of EDA’s potential in the government sector involves what is going on (and not going on) in sharing information about terrorists among the multiple agencies of the Department of Homeland Security. To keep it simple, let’s hypothesize that there are two databases of suspected terrorists in the U.S. Government. One is at the CIA and the other is at the FBI. (Just to be clear, we are working up this example on a purely theoretical basis to illustrate a point using a situation that most of us can appreciate. The true nature of these types of databases is beyond our specific knowledge.) As we have read in the news, there were historically some problems in coordinating the sharing of information about terrorist suspects between these two organizations.
Although some of the problems might be organizational, a good measure of the issue is technological in nature. The two sets of data stores are likely to be running on different platforms, written in different languages, and utilizing incompatible labeling methods. For example, the FBI might categorize terrorist risk profiles on a scale of 1 to 5, whereas the CIA might categorize them using letters A, B, and C. The two agencies might want to exchange data, but the technological and data labeling challenges make it complex and costly to do so. On top of that, there are great security concerns about who can see the data on each side, and so on.
Now, at the risk of repeating ourselves, we know that it is possible to integrate these two systems so they exchange data using proprietary middleware. However, it is going to be too costly and slow to really work. Using an EDA approach based on SOAP Web services, we could establish some event services (more on this in Chapter 5, “The SOA-EDA Connection”) that produce events whenever a terrorist profile is updated or created in each system. Without preconception or maintainability issues between them, the two event services (event producers) at the FBI and CIA, respectively, could feed valuable data into a central message hub for use in any number of EDA processors.
Adding to this setup, let’s say that the Transportation Safety Administration (TSA) wants to be able to plug in the names, or some other unique identifier, of airline passengers and match them against known aliases of high-profile terrorists. The inquiries from the TSA consist of a stream of events that asks “Name_match?” for each passenger name. The EDA processor can pull together these inquiries with the streams of events, and patterns of past events, from the FBI and CIA databases in real time. This setup is shown in Figure 4.6.
Furthering the example, we might also want to correlate if any of these passengers who have names that match known terrorist aliases have paid for the tickets with a debit card, or are flying one way, and so on. To accomplish these follow-on objectives, we could set up a complex event processing application that listened to events from a number of separate entities, including the banking system, the NSA, and so on. As shown in Figure 4.7, the event processor could be programmed to look for the matches, and flag them as suspicious passengers. The whole key to this working is the ability of the EDA to view multiple event streams in real time. This is only possible to do with any kind of efficiency or economic justification using EDA.
Figure 4.7 Event listeners can integrate numerous systems that might be entirely separate from one another, forwarding event data to a complex event processor. In this example, event data from five different agencies and private entities is examined for patterns that would indicate a suspicious airline passenger.
The ability to synchronize the analysis of multiple data streams in real time is a great potential boon to health-care researchers and providers. Whether it involves correlating cancer treatments with genotypes worldwide or figuring out which brand of bedpan breaks the most, research of health-care practices is an urgent task for a large number of stakeholders. The health-care industry typically wants to provide good care at an ever decreasing cost. Insurance companies, drug companies, doctors, patients, labor unions, private employers, the government, and the academic world all have a stake—and a say—in how health care is managed and paid for. EDA-based approaches to IT offer value on many fronts in this regard.
If we convert the complex EDA approach we used for the government in Figure 4.7 and applied it to the health-care field, it might look like Figure 4.8. If the integrations between the players in the health-care industry are achieved through a message bus pub/sub approach, event listeners can be deployed to examine the event streams that occur within the conduct of daily business among the players. Without any dedicated integration, then, a complex EDA can look for patterns that are relevant to specific lines of inquiry. For example, from the EDA, it could be possible to detect abuse of prescriptions, the success of treatment protocols, or prescribing patterns of specific doctors.
Compliance is the other area where EDA has the potential to make a positive impact. This is a vast subject, one on which an entire book could be written. However, for our purposes, the EDA-related benefit to concentrate on with compliance is the paradigm’s ability to enable continuous monitoring of separate activities, which is often the key to compliance. Simple compliance, the requirement that a single set of activities, such as financial accounting controls, be monitored and verified, is itself a complex IT challenge. Multithreaded compliance, which requires that multiple independent systems or even multiple companies and government organizations all provide visibility for audit and control, can be so complex as to render it cost prohibitive.
For example, a global corporation must comply with a host of separate financial disclosure laws, environmental laws, and privacy laws worldwide. The simple act of exporting an employee’s personal address from a European data center to an American one could trigger a criminal charge for violation of the EU’s strict privacy laws. How can a company be sure that it is not violating such laws? Add to this the distributed and independent nature of certain corporations worldwide and you can see the enormity of the challenge.
With an EDA approach, it is possible to set up event listeners that can examine event data relating to multithreaded compliance activities and alert auditors. In our employee privacy example, the data export activity of the employee databases in the EU can be monitored for events that signal that an improper data export has been initiated, or requested. The event processor can then issue a warning to the administrator or the user that the data export violates both company policy and the law. Or, the event processor could be programmed to block the export itself. In this way, EDA provides a dynamic, cost-effective, and flexible way to achieve the monitoring necessary for global multithreaded compliance.
In this chapter, we looked at several of the areas in business and society where EDA has the potential for positive impact. The following are places where EDA has a strong potential for positive change:
Within enterprise computing itself, at the infrastructure and data-center operations level, EDA has the power to enable improvements in real-time system monitoring and security monitoring, two areas where conventional integration techniques can be slow to set up and modify, and costly to maintain.
EDA has the potential to bring about changes in application integration, beyond even the benefits wrought by Web services and SOA. The level of flexibility potentially enabled can have an impact on the way that data centers support application infrastructure.
The capacity to enable complex composite applications is both an innovation of EDA and also a challenge to data-center operations. As applications become increasingly detached from specific servers and distributed across multiple boxes, data centers, and even organizations, the IT team must understand and respond by modifying their support, testing, and deployment processes.
Overall, EDA results in enterprise agility. The paradigm helps make real-time data more readily available, which has the ability to guide decision makers in strategic planning. Operationally, EDA’s inherent flexibility makes possible the kind of rapidly cycling changes in business that strategic agility demands.
In the broader societal context, EDA has the potential to change the way that government and the sciences use computing to accomplish their missions. For example, antiterrorism programs can be greatly aided by the rapid, simultaneous visibility into multiple criminal and commercial databases worldwide.
In terms of compliance, a major stress for large companies, EDA enables cost-effective and dynamic continuous monitoring of separate systems. This is a boon to auditors who need to set a level of assurance that multiple threads of global legal requirements are being followed.