Appendix 3
Why Has SOA Failed So Often?
,
A.3.1. The need for flexibility
Facing competition that becomes tougher day after day, many large companies have struggled to improve the flexibility of their business processes and thereby gain an advantage over their competitors. Aligning the information system to business processes in order to decrease the “time to market” became the new motto. Service-oriented architecture (SOA) has been one of the most advocated solutions to meet this flexibility requirement. As a matter of fact, the various service-oriented approaches have attempted to solve two separate problems: the first is that of reusing existing components and the second is the flexibility of business processes. In Section 4.2, we presented various forms of reuse as solutions to the flexibility of requirements but obviously there are situations in which reuse is justified independently of any need for flexibility. Applying simplicity through reduction or simplicity through organization could indeed be legitimate enough reasons for reuse. Opening the system to the external world can also be considered a specific form of legitimate reuse.
As was discussed in Section 5.2, three layers are traditionally considered in software architecture: the data access layer, the service layer, and the user-interface layer.
Figure A3.1. SOA amounts to splitting the service layer into a business-process and a processing layer
Roughly speaking, introducing SOA into a classical three-tier architecture amounts to splitting the service layer into two sub-layers: a business-process layer and a processing layer. The processing layer is responsible for implementing various business rules. It thus brings, essentially, reuse. The business-process layer, on the other hand, brings flexibility by allowing, at least in principle, recombination of services from the processing layer in different ways.
In fact, this approach has only rarely kept its promises, at least as far as flexibility is concerned. Quite on the contrary, it has often increased complexity so much as to freeze the system in a state where no further change was possible. We provide below our interpretation of this failure.
A.3.2. First issue: no suitable enterprise architecture
There are three different types of processes in a company:
– The business processes such as taking orders from a customer, shipping an order, delivering a product, subscribing to a loan, and booking a trip.
– The management processes concern making decisions and coordinating actions within a company.
– The support processes such as recruitment and payroll. These are unrelated to the core business and are common to many companies.
When we speak about flexibility, this only concerns the first category mentioned above, namely the business processes on which we will now focus. Among these, many require no flexibility. Placing an order for a product that needs no specific configuration is an example. These static processes are those that largely define the enterprise architecture (which we will define shortly). Flexible processes, on the other hand, could benefit from a service approach, which allows rearranging elementary business services. Taking out a loan with an insurance company or booking a trip using several partners of a travel company are examples in this category.
Unfortunately, many companies have strived for flexibility too early, before creating a robust foundation for the well-established and stable processes. The immaturity of existing architectures has caused IT teams to invest much of their time and energy in maintaining an infrastructure for perfectly stable processes. This, in turn, has prevented any serious and reliable implementation of flexibility. This situation was diagnosed in the famous paper “Avoiding the alignment trap in IT” that we mentioned earlier in Section 3.3.3.
A.3.3. Second issue: no data integration
Among companies that have attempted to implement an SOA approach, many faced the data-integration problem. By data-integration issue, we mean the necessity to define a consistent and companywide semantic and syntax for a significant set of business terms. Terms such as “customer”, “subscription”, and “product” must necessarily be harmonized before they can be used by services, which are shared among different business units. In large companies, as no single individual alone can master the full range of concepts and terms, a phase of modeling is necessary. The point here is that these modeling tasks can be extremely complex and heavy. They have been largely underestimated most of the time, which has thus prevented a full-scale implementation of the SOA approach.
As a matter of fact, the magnitude of the task, which requires defining a consistent business-object model for a large domain of activity (banking, insurance, telecom, distribution sector, or other industry sectors), is so important that it is out of reach for even the largest IT departments. This is more an R&D task that has to be undertaken by an industry sector as a whole1. This approach is, in fact, nothing but a higher order of reuse that still has to gain wider acceptance if SOA approaches are to become more than consulting pipedreams.
A.3.4. Identifying the operating model
What is traditionally called the enterprise architecture has two facts: one is the logical organization of business processes and the other is the IT architecture supporting these processes. Starting from the above critique, regarding flexibility, we realize that the first aim is to build a robust and perennial enterprise architecture. For this to be possible, we should identity one invariant that characterizes the way a company does business in the long run. We shall follow the work of Weil et al. here (see [ROS 06]) and introduce the concept of an operating model. One obvious first guess for such an invariant could be the business strategy of the company. The business strategy covers things like the choice of a growth mode, the relevance of investing in specific markets, or developing a particular product. The authors argue that the business strategy is definitely not an invariant, as it is likely to change with new market opportunities or constraints.
A number of studies from MIT have suggested that the appropriate invariant is instead the operating model, which is defined as follows.
Definition: An operating model for a company is the optimal choice, for producing goods and services, of two parameters:
1) The appropriate degree of standardization of business processes across the various organizational units of a company.
2) The appropriate level of data integration on which these processes are based.
Let us, in turn, define these two concepts a little more precisely.
A.3.4.1. Data integration
This refers to defining a consistent set of business terms and appropriate technical formats (Java interfaces, XML schemas, etc.). The integration can be performed either within a process (intra-process integration) or across processes (inter-process integration).
The question to ask to evaluate the optimal level of data integration is:
How much does a business transaction in one business unit depend on the availability, timeliness and accuracy of data in other business units?
A high level of data integration will facilitate the exchange of information between applications and services as no complicated conversion will be required. Within a process, changing the scheduling of services can be considered. If data integration is inter-process, a service from one process can be added to another process. Flexibility, intelligibility, and transactional integrity are all promoted by a high level of data integration.
A low level of data integration has other advantages. As we have already mentioned, achieving data integration on a large, inter-process scale is a very demanding and expensive endeavor. Therefore, the most obvious advantage of a low level of data integration is simply the savings gained by not engaging in a long and risky integration process. A second advantage is that of the relative autonomy of each business unit regarding the definition of its data, something that can favor taking initiatives more easily.
A.3.4.2. Process standardization
Standardizing processes implies that each organizational unit will execute processes the same way.
The question to ask to evaluate the optimal level of process standardization is:
What is the benefit for the company of having all its business processes executed the same way in each of its organizational units?
A high level of process standardization fosters a lower variability in the execution time of processes. The overall predictability is thus improved.
A low level of process standardization, on the other hand, avoids the local rigidity that standardization implies. It also avoids the situation where some processes, which are locally perfectly efficient, are thrown away and replaced with global ones. Local creativity and initiative is favored.
A.3.5. Which models are compatible with SOA?
It is important to realize that neither low nor high levels of process standardization or data integration are intrinsically good or suitable by themselves. Rather, each company should identify which operating model best fits its own way of doing business. Weill and Ross distinguish four main operating models, depending on the importance assigned to high or low levels of data integration and/or process standardization. They are distinguished by four names and depicted on a two-dimensional graph.
Each operating model is adapted to a different business context. Below, we briefly describe each operating model and discuss whether it is suitable for a service-oriented approach.
Figure A3.2. The four operating models are defined using two axes: vertically, the degree of data integration and horizontally, the level of business-process standardization. A prerequisite for SOA to make sense is that data integration is high
A.3.5.1. Diversification model
This corresponds to weak data integration and a weak standardization of processes. It is typical of companies that share few customers and few suppliers across business units. Each business unit in a company with a diversification model runs its business essentially independently from the others. Each proposes its own services to its own customers.
For these kinds of companies, innovation and independence of decision are key factors of success. They would be largely penalized if they had to comply with the semantic rigor and the discipline imposed by data integration and process standardization. Migrating to a service approach would most likely imply costs incommensurate with hypothetical benefits.
The service approach could make sense locally, but only to couple systems living in different technological environments.
A.3.5.2. Replication model
This corresponds to weak data integration and a high standardization of processes. Companies that fit this operating model leverage their ability to innovate on the global level by introducing new processes relatively quickly. Each organizational unit has a relatively large autonomy but must nevertheless comply with a large set of companywide processes. The main expected source of profit is precisely this quick, companywide implementation of new processes.
Among the four operating models, this is probably the one that least fits a service approach.
Data integration is low to start with and increasing it would imply important costs for a goal, process flexibility, which is not needed anyway, as processes are designed in a centralized way.
A.3.5.3. Coordination model
In such a model, data integration is high but there is no standardization of processes. Concepts such as “customers”, “suppliers”, “products”, “services”, and “partners” are shared among the different organizational units. Some basic business processes are integrated on a global scale, but, essentially, each business unit designs its own process. The possibility of building a fully integrated customer service and the ability to track a product throughout the supply chain are the main benefits of data integration. Regarding processes, each unit can be creative. Achieving low costs is not the main concern for these companies, whose strategy is more to propose the best possible services to their customers.
Among the four operating models, this one is no doubt the best-suited to a service approach.
The essential data, customers, products, and suppliers are already largely integrated and ready for use by each service. Moreover, the high variability of local processes fits well with the main advantage provided by a service approach.
A.3.5.4. Unification model
Companies that correspond to this operating model use standardized processes based on highly integrated data. There exists a strong coupling between the different organizational units and their autonomy is not a high priority. They strive to provide the optimized services to their customers by minimizing the variability of processes. They often use integrated supply chains, which creates strong dependences between organizational units. The management is centralized; it tries to optimize growth through economies of scale. This model is particularly well suited for companies, which sell commodities.
The fact that data is integrated allows considering a service approach. However, as most processes are designed in a centralized way:
This operating model does not really fit into the logic of a service approach, which could nonetheless benefit, locally, to some specific processes that are not part of the core business processes.
A.3.6. Conclusion on SOA
We can identify two main reasons, which explain why many SOA projects have failed prematurely:
– Many companies did things in the wrong order. Rather than building a robust and scalable architecture for their stable, core processes, they tried to achieve flexibility prematurely. In the most extreme cases, this led to the so-called alignment trap, namely the IT department spending most of their energy and money on maintaining an architecture that never had the time to become mature because of premature attempts to align IT with ever-changing business requirements (see Section 3.3.3). It should be emphasized that an
SOA architecture, if useful at all, should only be considered as a last step in any enterprise-architecture maturity model. Robustness considerations should come first.
Among companies that tried to implement SOA, many did not need it in the first place. Companies that do not benefit from flexible business processes have no reason to switch to a service approach, as this will incur heavy financial and organizational risks for an often unpredictable outcome. Sorting the operating models according to their relevance for an SOA approach we have: (1) replicated (worst), (2) diversification, (3) unification, and (4) coordination (best suited for SOA).
It remains true, however, that, independently of any flexibility considerations, a service approach can still be partly motivated by reuse considerations, as an implementation of simplicity through reduction and simplicity through organization principles (see Section 2.2).
1 The telecom industry has been pioneering this approach with its TM Forum' Information Framework (SID) (see http://www.tmforum.org/InformationFramework/1684/home.html).