Businesses have been focused on process improvement to improve operational efficiency since the era of business process reengineering (BPR). However, BPR failed in many organizations because the supporting information systems did not allow changes without significant rework. Process integration was born out of a need to allow business processes to be quickly automated and adjusted. Furthermore, it has expanded to allow for processes to be managed and monitored in real time providing a feedback loop to business management. Management dashboards provide graphical views of critical business processes, events, and key performance indicators, and are updated in real time.
Process integration is a powerful tool for enabling managers to quickly react to important business events and to be proactive when opportunities arise. However, in order to reap the rewards of process integration, companies need to invest in defining, managing, and optimizing their business processes. This is still a large leap for the majority of companies.
Process integration is an application that sits on top of the underlying applications that actually perform the business functions. The process layer defines, monitors, and manages a business process that spans multiple applications. The difference between process integration and either application or information integration is that the integration is defined as a process rather than as a message between applications or as information. Composite application integration and process integration are similar in many regards. However, the three key differences between process integration and composite integration are that process integration is focused on the definition and creation of business processes rather than code, is aimed at business process analysts versus programmers, and does not presuppose a service-oriented architecture.
Process integration uses a variety of technology in application and information integration including adapters, brokers, etc. to integrate with the underlying applications. Although process integration technology could conceivably be used to create composite applications by tying together business services and components, standardized interfaces and business components and services are not a prerequisite. Furthermore, the objectives and developers of process integration and composite applications are different. As previously stated in Chapter 12, composite applications are usually created by application developers and the goal is rapid application development through reuse. Business analysts more often define process integration solutions and the goal is improving business efficiency by automating business processes and managing them in real time.
The process integration market segment is predicted to grow considerably in the next few years. This growth is likely to be fueled by integration solutions or process templates that provide a large percentage of the solution. Compliance regulations in finance and healthcare are driving adoptions of solutions based on integration technology. Process integration provides an excellent platform for these solutions, because it makes them easy to customize. At best, any packaged solution will meet 75% to 85% of needs. A process-integration template can reduce the time and cost of customizing solutions. Once the technology comes through the door with the industry or compliance solution, then it can be used on other projects. Case Study 13.1 shows how process integration accelerated the integration of business acquisitions and enabled flexible business processes across divisions (Gold-Bernstein 2002).
Process integration can help companies optimize business processes, reduce operational costs, and improve business agility. It provides business and operational managers the tools they need to stay on top of a real-time business environment. But the technology alone does not guarantee success. To achieve sustainable competitive advantage, companies need to commit to process management and continual process improvement.
Process integration technology is well suited to the following business requirements.
Process-improvement initiative. It has been shown that companies that focus on process and continual process improvement outperform their competitors to a very significant degree (see Chapter 9). These companies tend to be the market leaders in their industry. Continual process improvement delivers competitive advantage, but requires high-level commitment and investment.
Industry-specific processes. BPM provides excellent technology for creating templates to implement specific business processes. For example, a risk management system for finance might integrate information from multiple systems to calculate risk. Industry solutions provide the majority of the solution and reduce implementation time and risk of failure. The process integration technology makes it easy to customize the solution.
Compliance solution. Process integration vendors have focused heavily on compliance solutions because they provide a captive market. Healthcare companies are required by law to implement electronic transactions formatted in the HIPAA standard. Although compliance usually represents a cost, healthcare organizations are finding that they are actually achieving an ROI from HIPAA solutions. Electronic transactions, combined with back-end integration, lower transaction errors and costs.
Sarbanes-Oxley is another compliance solution driving the adopting of process integration technology. It requires executives to certify the process that generated the financial figures, such as how revenue is recognized, and holds executives personally responsible for signing off on the financial statements and criminally liable for any inconsistencies. The image of high-level executives going off to jail in handcuffs is a powerful motivator for executives to be interested in technology solutions that will protect them. Compliance also requires limiting any variance from the proscribed process to ensure the process has been followed. Process automation and integration is well suited to providing these capabilities and providing the platform for compliance solutions.
Process integration technology includes business process management (BPM) tools for automating and managing processes, workflow management for manual processes, and collaboration tools. There is a great deal of convergence in the market. Just about all the EAI vendors include process level integration. There are also a number of pure-play BPM solutions that include adapters for back-end integration or integrate with an integration broker or enterprise service bus. The workflow vendors have entered the market, as well as the enterprise content management (ECM) vendors. A couple of rules development platform vendors have also reinvented themselves as BPM vendors. In fact, just about every vendor with a process or rules engine has entered the market. The different vendors offer solutions optimized for different types of processes.
Business process technology typically includes a modeling tool for defining the business process. The standards for a process modeling and execution language are evolving (see Chapter 9), and most modeling tools are proprietary. However, many vendors are at least giving lip service to the emerging standards, especially BPEL. This will be important for model portability. It enables a company to leverage the investment in defining business processes and exporting the execution code to different tools. At a minimum, if the tool can import and export BPEL, it will prevent technology lock-in, and ensure long-term viability.
Many process vendors are now offering both process and rules engines. Process engines are designed to manage the state of a process, including long-running processes, which can take hours, days, or even weeks. For example, a cable or DSL line may be installed days or weeks from the initial order, and could involve third-party subcontractors, such as the installers. The system needs to track the state of each task or sub-process in the overall end-to-end process over a period of time. This has implications for transaction management in long running processes. A transaction committed weeks ago cannot necessarily be merely rolled back. It may require a series of compensating transactions to “undo” the action, such as sending a letter out to the customer.
It is becoming more common for vendors to also offer a rules engine to define the parameters that control the process. Rules are much easier to change than code embedded in a process execution language or a process model. Furthermore, rules can be distributed and localized. If the process spans organizational entities, then parts of the process are often managed by different entities. A rules engine enables the rules specific to a part of the process to be managed and changed locally. For example, a rule might say that orders over $1,000 require management approval. In very busy times, this rule might be changed to $2,000. The process does not need to be changed at all. Process rules can be localized by departments or geography. Rules distribution and federation puts control in the hands of the line-of-business managers. Because of the flexibility rules provide, it is becoming more of a necessity for process integration.
When choosing process technology you must first determine what type of processes are predominant in the organization. For example, some process tools were built primarily for automated processes, and have limited support formanual processes. If you are attempting to reduce the inefficiencies in manual processes, then look for a solution that offers strong workflow capabilities, including workload balancing among employee roles and an interface to display individual work assignments. It is beneficial to have both manual workflow and automated business processes supported with a single modeling tool as it significantly lowers training and maintenance costs. But processes that predominantly involve people, especially large numbers of people, will require robust and scalable solutions.
If the implementation is part of a business process improvement initiative, look for strong modeling and simulation capabilities, including time and cost-based analysis of tasks and processes. Simulation tools provide historical analysis based on past process performance along with what if analysis. Success also requires a commitment to process management and continual process improvement. In this case, the tool is only as good as the people who use it.
When seeking to gain real-time visibility into business processes, BAM is important. The BAM market is still in the very early stages. EAI vendors, BPM vendors, software management vendors, pure-play start-ups, and infrastructure-management vendors are offering BAM solutions.
Gartner, which coined the term, defines different stages of BAM. The first stage is informational. The management dashboard provides real-time visibility into business processes and exception alerts, then a person needs to go and decide how to respond to the alert. When business intelligence is added, the alert includes analytical information to help produce a more informed decision more quickly. The future of BAM is to have the system automatically respond. IBM Research is working on autonomic computing with the goal to create computer systems that regulate themselves, the way our autonomic nervous system works in our bodies. Most of the BAM technology currently available is at the level of providing real-time information in management dashboards.
Collaboration software, also called groupware, has been around since the mid 1980s. However, it is now starting to become part of the integration infrastructure. Collaborative process integration includes facilities to allow disparate team members to communicate, coordinate, and manage the process, as well as collaborate on documents, models, and other project deliverables. Collaboration solutions are generally network-based solutions, and may integrate with enterprise content-management solutions and desktop tools. Collaboration tools are also being introduced in supply chain and manufacturing environments, where organizations need to collaborate and negotiate on product design and other issues.
Large organizations are likely to have different types of processes to support. The issue is whether the company should choose one process solution and force fit it to all applications, or deploy multiple technologies and optimize each solution. Unfortunately, the latter approach will guarantee redundancy in modeling tools and process engines, which ultimately impacts the bottom line and increases the total cost of ownership. The optimal solution would be to have a complete service-oriented architecture and deploy integration services as needed. The vendors, however, have a vested interest in their own platforms, and this is not yet a reality. Companies may have no alternative but to support several different process integration platforms.
This specification provides implementation guidance for the development of a process-integration based solution. It is most likely that the process design specification from Chapter 9 will form the basis for the implementation.
Process integration represents a specific style of integration. It manages the integration of an end-to-end business process, as well as processes that span organizations. It is appropriate for business process improvement initiatives, automating interactions between organizational entities, and implementing industry and compliance solutions.
This section describes the specific technical problems that are being addressed in the implementation, and provides context for specific implementation. See Appendix K for the full specification.
The scope of a process integration specification is limited to the business processes that are being automated and integrated. It can include processes within departments or between departments, divisions, territories, companies, and business customers and partners.
This section identifies all stakeholders in the implementation, including business managers who control all or part of the process, system designers and architect(s), and the development team who will execute the implementation. Because there may be multiple managers responsible for parts of the process, continuous process improvement, and process optimization will be difficult to achieve without a process owner or manager who is responsible for the end-to-end process. Any other participants or stakeholders should also be identified, including their roles.
Process integration is in the very early stages of adoption, and patterns may emerge over time. This section includes the following patterns:
This section defines the particular pattern that is being used and provides details on the configuration of the specific components of the implementation.
Process automation requires underlying application integration services. The solution may provide integration services such as adapters, or may integrate with an EAI solution. Many BPM tools do both in order to provide implementation flexibility.
The services included in process automation include process modeling, process simulation, process management including support for long-lived processes, business rules management, transaction support including compensating transactions, as well as all necessary application integration services. Because it is important to manage process models as valuable enterprise knowledge and content, Figure 13-1 includes a process repository. However, this can be part of an existing repository or content management solution. Note that BAM is on the diagram because process analytics and management dashboards are often part of the process automation solution. However, it can also be a separate implementation. A good example for process integration is an employee review process. This type of process is often difficult to manage, requires integration across a diversity of data sources, involves a lot of people, and is often labor intensive. Business process integration could be kicked off at a time indicated by corporate policy and procedures. The information necessary is collected from a variety of sources and the templates for the review are created and forwarded to the appropriate managers.
The Process Automation Implementation Table (Figure 13-2) specifies the integration services included in a process automation solution, along with relevant implementation details. Customize the following table for your implementation.
BAM provides real-time visibility into business processes. BAM can be part of a business process management solution or it can be a stand-alone solution. Because the BAM market is young and evolving, the current BAM solutions have different focuses and configurations. For example, some solutions focus on analytical information, monitoring data warehouses, and sending alerts when certain events occur or conditions arise. Some are more focused on the tools for graphically building the management dashboard. Some are focused on fast information access. The future direction of BAM includes automated responses and autonomic computing.
The basic services provided by BAM include a management dashboard with key performance indicators relevant to particular roles in the organization, and real-time analytics to populate the dashboard, which include either providing business intelligence capabilities, such as monitoring and aggregating events in data warehouse or integrating with another business intelligence solution. BAM solutions may include information integration capabilities to pull information from analytical data stores in real time. The actual monitoring capabilities can be implemented with application agents, process event logs, or it might be part of the BPM solution. A BAM solution may also contain modeling or development capabilities to define the BAM solution, as each one is likely to be unique. A good example of using BAM is the management of the entire process of admitting patients in hospitals through discharge and might even included the coding and billing process. Hospitals are a complicated environment and by monitoring all aspects of the process, actions can be taken to deal with especially heavy workloads, dealing with problems in the coding process or managing the accounts receivables and collection processes.
The reference architecture in Figure 13-3 (page 254) depicts the basic services that might be included in a BAM implementation with current technologies. The architecture is likely to evolve significantly as the market matures.
The Implementation Table (Figure 13-4, page 255) specifies all the integration services provided in the BAM solution, along with relevant implementation details. Modify the table on page 255 to depict your implementation.
Collaborative process integration provides a platform for integrating work from different team members in different locations. It allows team members to manage their work process, communicate, coordinate, and collaborate on documents, models, and other project deliverables.
The heart of the collaborative solution is a collaboration portal that provides access to all information and services supporting the project. This includes artifacts including deliverables, meeting notes, discussions, decisions, etc. A project repository is important for managing project artifacts. The repository can be part of the collaboration platform, or may integrate with a content management solution.
Version control with check in and check out facilities are crucial features for collaboration. Often check in/check out is part of the collaboration platform. However, it is also possible for the collaboration portal to integrate with a content management system or a version control system. The latter is important for software development as the code is often managed by an existing version control system. There is no either/or here. A number of different configurations are possible.
Project management may also be provided by the platform, or it can integrate with a project management system, such as Microsoft Project. Workflow is often included as part of the platform to manage the collaborative process. Project coordination is provided with an electronic calendar and meeting scheduling. A facility to support threaded discussions and negotiations, and document decisions is also helpful. Because much project communication is done through e-mail, which is particularly difficult to manage and archive, e-mail integration is becoming more important. The collaboration platform should include role-based security with granular levels of authorization, so access rights can be defined at the artifact level. For example, some documents may require the input from the whole team, and some may be “read only” for most of the team, save a couple of members.
As noted previously, although groupware solutions have been around since the mid 1980s, they are beginning to provide more integration capabilities as they evolve and integrate with other enterprise solutions. Collaboration is also becoming important in some other integration patterns, especially B2B integration, where partners need to collaborate and negotiate on business processes, product designs, and other joint efforts. A good example for collaborative integration is any design process that involves outside vendors. The automotive industry has been a leader in applying this technology. The collaboration tool provides version control on designs, and tracks conversations, feedback, and decisions.
As process collaboration integration becomes more firmly entrenched in organizations, the architecture is likely to evolve. Figure 13-5 is a reference architecture based on a collection of capabilities currently available, although no one vendor may offer all of them.
The Implementation Table (Figure 13-6) specifies all the integration services provided in the collaborative integration solution, along with relevant implementation details. Modify the following table to specify your implementation.
Make an organizational commitment to continuous process improvement. Process optimization is a discipline and a journey, not a single software installation. Although it holds the potential to deliver competitive advantage, it requires significant investment. Organizations that create a culture of continuous process improvement will reap the highest rewards (but this requires the highest level of commitment).
Measure process performance. Measurement is essential for process improvement. Process performance can be measured in terms of time and/or cost. It can also be measured in terms of overall business performance with metrics such as profitability, customer satisfaction, or growth in market share. Simulation tools are extremely helpful for calculating performance from actual process statistics, and doing what-if analysis.
Reward process improvement. You can expect employee behavior to be tied to how performance is measured. To create a culture of continuous process improvement, reward process improvement. Otherwise, there will be little incentive to embrace continuous change.
Provide real-time management dashboards. Put business managers in the driver's seat with customized dashboards that display key performance indicators and real-time alerts. Provide business managers with the information they need to proactively manage the business. Give them the ability to easily manage and change the processes and business rules they are responsible for.
Manage an automated process as a reusable component. Automated processes can be reusable just as any component or interface can be reused. By allowing processes to be pieced together in new workflows, a business can quickly optimize its operations as changes occur or insight is gained through activity monitoring.
The nature of process integration is that it is never done. The first implementation is just a starting point that will be adjusted and changed. The next step is to ensure that the changes are being tracked and managed so that the processes are defined and documented.
In addition, enterprise integration is an ongoing journey. The integration infrastructure will evolve over time and the process integration components are only a part of the overall solution. It needs to be refreshed to stay in step with the overall architecture. The integration roadmap (Figure 13-7, page 260) and life cycle is one you will return to again and again.
Each new project will have its own unique requirements. An implementation may be a combination of integration styles and patterns. Use the patterns and guidelines provided in this chapter and book as a starting point. Available technology choices may constrain the ability to implement the optimal architecture. It is also important to remember that the patterns and technology will change and that process integration is one of the least developed of the forms of integration. To see an example of service-oriented enterprise integration reference architecture that can accommodate any and all integration needs, see Chapter 14, Conclusion: Best Practices for Enterprise Business Integration.