Chapter 2

Complexity, Simplicity,
and Abstraction

,

 

 

 

Recursion is the root of computation since it trades
description for time.

Alan Jay Perlis — Epigrams on Programming

 

2.1. What does information theory tell us?

We start our journey through complexity and simplicity concepts with mathematics or, more precisely, information theory. This might seem an exotic topic if what we have in mind are applications to the IT world. However, concepts that will be at the core of our future preoccupations, information, randomness, and especially complexity have all been under close scrutiny by mathematicians for over more than half a century now. In their hands, these concepts have evolved into a set of ideas, which is both deep and robust. Moreover, information theory is actually one of those few areas where mathematics succeeded in rigorously formalizing imprecise, almost philosophical concepts, such as complexity and information, to which they bring a unique insight. It would thus seem unreasonable for us to overlook this body of knowledge altogether. These information theory concepts form a collection of metaphors that will help us build a healthy intuition that will prove helpful later when we venture into less rigorous but more practical IT concepts. As we shall see, this first look at the subject, through mathematical glasses, also highlights a number of important issues and limitations, which occur as soon as one seriously attempts to define complexity.

As information theory is a highly technical and abstract topic, we can barely afford here to do more than just scratch the surface. We shall strive to present in plain language the major findings in information theory of relevance to us. The interested reader will find more details in Appendix 1.

Our quick overview of information theory will focus on only three concepts: Shannon's entropy, K-complexity, and Bennett's logical depth. Assume for simplicity's sake that any object or system, whose complexity we wish to define, is described by a binary sequence s such as 001101110… The three concepts mentioned above have one important point in common: they all evaluate the complexity of a system as the quantity of information that its description s contains, assuming that we have a specific goal in mind for s. This goal, as we shall see, is a foreshadowing, in the restricted mathematical context, of the concept of value that we shall examine in Chapter 3. To make things as concrete as possible, we start with an example drawn from classical computing [DEL 99].

Suppose we want to assign a complexity to the compiled code of an algorithm that solves a certain class of problems. The system under consideration here is thus a piece of software. The binary sequence s that describes it is simply the compiled code. The goal we are trying to reach when compiling the source code to create s is something like this:

– First, we would like the compiled code to run as fast as possible. Thus, compiled code that runs fast is certainly of higher value than slower code.

– Also, we would like the compiled code to be as short as possible. A program with a low memory footprint will thus be of high value. The latter condition is in this day and age less important than the first one. Compared to the first criteria, we might thus give it a lower weight in our evaluation of the complexity of s.

– Finally, we can consider that the larger the set of problems the algorithm is able to solve, the higher the value of the compiled code will be.

To this list of quantitative features one could add still other things like the portability of the compiled code or its originality. In any case, the goal is some suitably weighted combination of all these quantitative and qualitative wishes. Altogether, they determine the value of the information or complexity enclosed in the compiled code s. The price at which we would sell the code would probably be strongly correlated to such a goal.

From this simple example, we learn that one good idea could be:

For realistic goals such as the goal mentioned in the above example, there is unfortunately no full-fledged mathematical theory available. However, for simpler but still significant goals, there is. We now briefly review the three of greatest relevance to our subject.

2.1.1. Shannon's entropy

Historically, the first concept of information to be formulated in mathematically rigorous terms was that of Shannon's¹ entropy. There are many different interpretations to Shannon's entropy. For our purposes and to provide a basis for further reflection, we shall present it first as a measure of the amount or randomness present in a system. Later, we shall briefly explain how this amount of randomness is related to an average quantity of information and also to an idea of complexity.

The idea is probably best understood by recalling the original context in which Shannon formulated his concept. Shannon was looking for an optimal method to compress information to transmit over noisy communication channels (an electric wire or radio waves). For this, he modeled a source of information such as a text to be transmitted, as a random generator of characters. We may indeed think of a large text in a given language, English or French, as being made up of a random sequence of characters whose probabilities are known in advance (the character “e” occurs more often than “k” in English, for instance).

The goal that we referred to in the previous section is here: “Encode strings of characters, whose probabilities are known, in such a way that their average encoded length is as short as possible”.

Shannon found an explicit solution to this problem, which we will not examine in detail here. The intuition behind his solution is, however, easy to grasp: it says, in a precise way, that frequently occurring characters should be encoded with short strings. These solutions are described, with minimal mathematical artillery in Appendix 1. However, for the moment, there are only two important things to understand here:

– The average length of encodings in Shannon's solution turns out to be larger if the source is more random (see Appendix 1 for an example). Shannon's entropy is, by definition, the average length of this optimal encoding. It is thus a genuine measure of randomness of the source of characters.

– The average length of Shannon's optimal solution turns out to be dependent on the probabilities of occurrence of each character only and not on the precise character set or alphabet that is being used. Therefore, Shannon's entropy has a significance that reaches far beyond Shannon's original question on finding optimal encodings. It is in fact a measure of randomness of any random event whose probabilities are known.

The relation of Shannon's entropy to an amount of information is obtained by rephrasing point 1 a little. When the randomness of a message is weak, we know ahead of time what the outcome will be. In other words, we do not gain much information when we learn the outcome of the random experiment. By contrast, if the source was highly unpredictable we gain a lot of information as soon as we learn the outcome that we could not have guessed in advance.

We postpone the relation of Shannon's entropy with complexity to the following section on Kolmogorov complexity, where it fits more naturally.

One last and very important property of Shannon's entropy is that it is very easy to compute once we are given the probabilities of the random events. Unfortunately, this ease of computation has led to numerous cases of misuse of Shannon's entropy, even when no genuine probabilities were available. Although there are few direct applications of this concept to IT; we chose to quickly present it because of its historical significance and because it makes a natural introduction to our next concept.

2.1.2. Kolmogorov complexity

In its original formulation, Shannon's entropy, as we have just seen, measures the randomness of a source of information that produces a random output (say strings of characters). But what if we would like to assign a complexity to a single object or system or string of characters? It turns out that there is such a concept. It is called Kolmogorov² complexity or K-complexity for short.

Its formal definition is somewhat technical, but its essence, as we shall see, is quite simple to understand. K-complexity is one possible measure of the amount of information contained in a binary string. The binary string could be the description of an object or not. It could be the sequence of characters of this book, a set of UML diagrams, the code of your browser, or the first digits of π. Suppose now that our goal is to compress this binary sequence as much as possible. Finally, we measure the length of this most compact form of the original string. Well this length is, in very rough terms, the K-complexity of the string.

As it stands, this intuitive definition immediately raises a number of questions. What exactly do we mean by compressing a binary string? Is there a unique and optimal way to do this? Otherwise, how would this definition make sense at all?

The formal definition of K-complexity requires the introduction of devices known as Turing machines. For our purposes, we shall consider them simply as conceptual versions of usual computers. Like ordinary computers, Turing machines run programs and output a result. Both the programs and the output are binary strings. The optimal compressed form of a binary string is nothing other than the shortest program that, when we run it on our Turing machine, gives us back our original string. If the string has a lot of structures in it, for example if there is pattern that repeats periodically like 011011011…, then the program that outputs this string can be much shorter than the original string. On the other hand, when the string is “random” and has no structure in it, then only an explicit output operation can produce our string. No compression is possible. The length of this output-operation program is just slightly over the length of the original string.

It can be shown (though we will not do it here) that the length of this shortest program does not depend, in a significant way, on the type of Turing machine we chose, at least when considering very long strings. This independence legitimizes K-complexity as an intrinsic measure of complexity. It is at the root of information theory.

2.1.2.1. Complexity of objects versus complexity of binary strings

It seems natural to define the K-complexity of an object or of a system as the K-complexity of its description in a binary form. Unfortunately, there is an ambiguity here because most physical objects or systems have various scales of description. Consider a large information system (IS). Describing the IS on a macroscopic level could be achieved in terms of use cases, for instance. On a more fine-grained scale, we could consider the source code of the applications as another valid description. Ultimately, on a microscopic scale, we could consider in principle the binary code running on each chip. Each of these descriptions contains a very different amount of information and thus we must select one to remove the ambiguity.

In section 2.1.4, we propose a definition of an abstraction level for the description of a system (think of UML modeling as a concrete example). The scale of description will be part of what characterizes an abstraction level.

2.1.2.2. Relation to Shannon's entropy

On the conceptual level, there is a link between Shannon's entropy and K-complexity that we shall not attempt to justify here but will mention as an interesting and profound result in information theory. It is best understood in the original context of a random string generator whose probabilities are known. If we were to ask: “What is the average of the K-complexities of these random strings assuming we use the probabilities of these strings as weights?” The answer is simple: it is (close to) the Shannon's entropy. In other words, Shannon's entropy in fact turns out to be an averaged version of the K-complexity. Disorder and complexity are thus related in precise sense.

2.1.2.3. Can we compute K-complexity?

Shannon's entropy is easy to compute. There is indeed a formula, which is given in the appendix. Unfortunately, there is no such explicit formula for K-complexity. K-complexity is important as a central concept. It allows us to speak rigorously about one type of complexity, and by comparing it to other types, it provides further insight. It is, however, of no practical use to evaluate complexity in concrete cases. Information theory proves that there is no, and never will be, any effective way to compute K-complexity. Here, effective means “in a predictable amount of time”.

It is important to understand that this impossibility has nothing to do with lack of imagination or progress in technology. It is a conceptual impossibility rooted in the limit of logic itself to obtain answers to some problems in a predictable amount of time.

We could be more modest and seek useful and computable approximations to K-complexity. There indeed exists such approximation but information theory brings us bad news once again: it will forever be impossible to evaluate the quality of such approximations.

Thus, even for such a simple goal as compressing the information as much as possible, we see that:

The only way out is to abandon the hope of finding a universal and computable concept of complexity and to look for complexity measures that are restricted to certain topics. Such examples will be discussed in section 2.3.1.

2.1.3. Bennett's logical depth

Random objects without structure have high K-complexity because, as we discussed, they are not compressible. With ISs in mind, we now realize that K-complexity is perhaps not such a good measure of the kind of complexity we are interested in. What if what we would like is to distinguish complicated systems, which are systems that have no short description, from truly complex systems, which require a lot of efforts to be produced? The table below summarizes the main differences we can expect between these two forms of complexity.

Complicated objects	Organized objects
Their description is long	Their description could be either long or short
Random objects are always complicated	Random objects are not organized objects
They are easy to produce	They require design or computing to be produced
They can be produced very quickly, and they even occur spontaneously	They grow slowly, and they never occur spontaneously

There is a concept, which formalizes this idea within information theory. It is called Bennett's logical depth. It again fits the scheme where complexity is measured as the amount of information with respect to the goal. For a binary string, this goal would be to “find the description that requires as little computation as possible to reproduce the given string”. Bennett's³ definition can be conveniently formulated in plain language if we use the same tools as for K-complexity. Suppose we have our Turing machine at hand and we found the shortest possible program p that outputs our string s. Bennett's logical depth is then defined as the number of steps of execution that p needs to produce s.

Rephrasing the above definition:

Similar remarks apply for K-complexity: Bennett's is as hard to compute and to approximate as K-complexity, unfortunately. When applying the concept to physical systems, a scale of description should be defined beforehand.

To gain some further intuition, let us look at some simple examples from the IT world using the above concepts as metaphors.

	Low K-complexity (short description)	High K-complexity (long description)
Low Bennett depth (no significant design work)	Standard piece of code that is generated automatically. Code for CRUD4 operations belongs here for instance	Databases with lots of unstructured information
High Bennett depth (significant design work)	Simple business rules (i.e. they are easily described) that need to be coded by hand because they are nonstandard (they require significant design effort)	Large and well-structured databases Large and well-structured code

Most ISs, taken as a whole, are both Bennett deep and K-complex.

2.1.4. Abstraction in light of scale and depth

An interesting by-product of the above discussion on K-complexity and Bennett's depth is that these concepts can help us clarify what is the level of abstraction of the model for a system. This is particularly important, as most complex systems, such as ISs, need several descriptions at different scales. As we will argue later, scale is precisely one of the attributes of an abstraction level of a model.

Having a clear idea of what an abstraction level is will be of particular importance when we discuss abstraction in software engineering in section 2.3.2 and IS modeling best practices in section 4.3.1. Abstraction levels in IT are customarily denoted by well-known terms that correspond, more or less, to layers of IT architecture such as the business process architecture, the software architecture, the application architecture, the hardware architecture, and so on.

To define the level of abstraction of a model of a system, we use an analogy. The system under consideration is represented by a binary string s: think of it as its description at a given scale as we explained earlier. A model of this system, at a given scale, can be considered a compressed version of its explicit description. We thus assimilate a model with a program p that generates the string s through some computing mechanism such as a Turing machine. The Turing machine should be considered here as a metaphor for the design process that leads from the model of a system to the system itself. To set the ideas in an IT context, consider the system as a business application, and its model is a set of UML diagrams.

What then is a good model? First, the model should be on an appropriate scale. If we want to describe business features of a business application, for instance (corresponding to s), we should model it using a set of UML diagrams that contain only this information and nothing else at a lower scale such as detailed technical information. Second, the model (corresponding to p) should not be too deep in the Bennett sense because this would mean that the amount of design necessary to go from the model to the finished system is very large. However, for models to be useful, they should not require too much additional work to build the system they describe. In other words:

Thus, the brief discussion given above suggests the following definitions for the abstraction level of a model of a complex system:

– The scale of description of a model is the first attribute of an abstraction level.

– The design effort needed to go from the model to the real system is the second attribute of an abstraction level.

We will have more to discuss on abstraction in section 2.3.

2.1.5. Harvesting information theory

Recall that we have discussed three different measures of complexity. From this point on, they will be considered metaphors that will, hopefully, add clarity to the various facets of simplicity, which will be defined in the following section.

– Shannon's entropy measures a form of complexity related to unpredictability of systems described by probabilistic means.

– K-complexity formalizes the idea that complicated or random objects have long descriptions.

– Bennett's logical depth formalizes the idea that organized complexity is ultimately related to a computing effort needed to produce an object.

We can perhaps argue that the above concepts are not much more than common sense. We think, however, that the fact that part of our common sense can be formalized into some universal concepts is quite relevant and even rather unexpected. These three concepts are to be considered cornerstone concepts that capture part of the semantic of complexity in its purest form.

Let us take stock of the main things we have learned on complexity from information theory:

– First of all, even in the very formal realm of mathematics, there is no single concept of complexity (or simplicity) that is relevant for all purposes. This will be all the more true for more practical definitions.

– The three examples have one point in common. They all formalize the intuitive idea that complexity is related to a quantity or value of the information contained in an object, a system or its description, relative to some previously set goal.

– Universal concepts of complexity are likely to be uncomputable in practice. Practical concepts of complexity, on the other hand, are likely to be applicable to very narrow classes of systems only.

– The uncomputability of complexity can be traced back to the impossibility of finding optimal descriptions of a system. This difficulty is inherent to the complexity and should not be attributed to a lack of knowledge or skill. Even if some care is in order here, not to draw improper analogies between mathematical results and the IT world, we can perhaps recognize in this intrinsic difficulty of finding optimal description, a metaphor of the difficulty of finding optimal descriptions of real ISs.

– There are at least two essential aspects of the complexity of a system. The first aspect is related to the length of its description. This is defined as the complication of the system. Random systems are always complicated. The second is related to the degree of organization that the system contains. This is called organizational complexity.

– Defining complexity in a sensible way always requires choosing an appropriate scale, or equivalently an abstraction level, for the description.

– Defining the abstraction level for the model of a system involves two things. First, defining what information is to be considered irrelevant at that particular scale. Second, what amount of computation or design the model encapsulates.

The following section will take on simplicity which, surprisingly, is much more than the opposite of complexity.

2.2. What does the design tell us?

Simplicity has become quite fashionable lately. The new motto “Less is more!” is gaining momentum in many fields, probably triggered by the issue of global climate change that prompts us to rethink our economic models to produce less and to recycle more. In the computing world, the most prominent examples are certainly those of Google and Apple products, which are well known for their simplicity. The former popularized minimalist web pages through its search engine page, which has a single-input field and one button. Actually, this minimalist solution became even more minimal recently when Google launched its instant search feature. Users have been freed from the remaining effort required to click on a search button; results are displayed on the fly, while the request is being typed. One-button devices, on the other hand, are Apple's specialty. The Apple mouse, the iPhone, and now the iPad are no doubt all milestones in a quest for simplicity in design. The last 10 years have witnessed a strong evolution toward simplicity; the era where stereos were expected to provide many displays, buttons, and tuning possibilities seems to be over.

Simplicity sells even better for services than for goods. When it comes to changing their ISP, their bank or phone companies, most people try to balance the often moderate increase in service quality, offered by a competitor, with the hassle of the paperwork generated by such a change. More and more companies have realized this and currently offer to take care of this paperwork for their customers. Even administrations seem to slowly get the point of simplicity. Indeed, large simplification endeavors have been launched in this direction in various countries, with moderate success so far, but there is still hope.

Getting back to our topic, we realize that an IS combines the two aspects just mentioned. First, most interactions with an IS are performed through user interfaces whose usability is directly related to the perception of greater or lesser simplicity experienced by users. Second, an IS plays the role of a service provider, which should supply information quickly, reliably, and is expected to be a substitute for traditional paperwork. Unlike an administration, though, an IS is expected to respond quickly to changing requirements.

While the various aspects of complexity of an IS are naturally associated with additional costs, simplicity is more naturally associated with creating value and opportunities, by focusing resources on what really matters. As we shall see though, they are deeply related and one often implies the other. The relationship between simplicity and value creation, on the other hand, will be the topic of Chapter 4.

Just as for complexity, which was addressed in the previous chapter, simplicity is something most of us would consider intuitive. We would no doubt recognize something as simple, whether we are considering a manufactured object, a business process, or a user interface. Defining simplicity, however, turns out to be a much more difficult task. There are a number of reasons for that. First, there is the polysemic nature of the word “simplicity”. This is actually no different from “complexity”, as we concluded in the last chapter. Second, simplicity is even more subjective than complexity, which can be measured and quantified, at least in special circumstances. Simplicity depends more on perception and is thus intrinsically harder to evaluate than complexity. Finally, it seems very hard to define simplicity facets that are truly independent. As we shall see, they often largely overlap and, at times, they are even conflicting.

These two things: polysemy and subjectivity, are what make simplicity both hard to define rigorously and, simultaneously, a useful and powerful guide in any design process, whether we are dealing with manufactured objects, business processes, systems, human organizations, or even concepts. In a thought-provoking little book [MAE 06], John Maeda, a world-renowned graphic designer and computer scientist from MIT, suggested 10 laws of simplicity. They certainly should not be put on the same footing as the mathematical definitions of the previous chapter because they do not have the same universality as the former ones. They are more like a summary of best practices distilled by an expert in design. Even though most of Maeda's insights on simplicity concern real-world objects rather than systems, we think that some of them can be adapted to our present concern, namely ISs.

Applying ideas of simplicity to computing and software development is not new. The principles of the Lean Software Development, which are adapted from the Japanese car industry, have been advocated by the Agile⁵ community for already some time now. Some of these principles overlap with ours and, when this is the case, we shall mention it explicitly. We consider, though, that from a conceptual point of view, John Maeda's approach is more insightful.

The various aspects of simplicity we highlight in the following sections apply to different categories of IT stakeholders. The three most important categories are:

– customers,

– in-house end-users, such as employees and business experts, and

– the IT teams in charge of maintaining and improving the system.

Keep these three distinct categories in mind when we touch on the relation between simplicity, complexity, and value in Chapter 3.

2.2.1. Simplicity by reduction

One obvious way to increase simplicity in any system is to remove useless or duplicate features. Truly useless features are rather uncommon in ISs. One commonly sees, however, duplicate communication systems or duplicate identity repositories (we do not, however, consider backup systems as duplicates). Such situations usually occur when a new system or a new feature replaces an older one without the latter being removed. This can end up creating twice as much maintenance work to perform. A situation that was initially meant to be temporary progressively develops into a permanent one. Maintenance or coordination tasks are then often done manually; they are error-prone and increase complexity without any real benefits, except perhaps in that a few people will be able to retain their old habits.

More significant, however, are situations where a genuine reduction in the number or scope of features (not just suppressing duplicates) could be beneficial. Indeed, keeping only essential features favors more focused work and increases the chances that users will fully master the tools they are using. Many applications are still designed to allow for a great deal of customization. The downside of this, is this flexibility and customization is that it is often attained at the expense of a thorough mastery of those tools by people who use them daily. Too often, the need for customization and flexibility is overrated, while the need for fully mastering IT-tools remains undervalued. We shall come back to these essential mastery issues in section 2.2.4. In short:

This motto applies to both IT and business users. Simplicity by reduction means:

– For developers: knowing a few technologies thoroughly.

– For business users: being autonomous on the range of tools they use daily.

Flexibility and customization can certainly favor high creativity in some cases but, too often, they just create randomness. Customization and flexibility features should thus really be kept only in those circumstances where creativity is clearly identified as a major factor for creating value.

A beneficial side effect of reducing the scope and the number of features is that those removed features are often also the most complex ones to implement and to maintain. In a way, all of these can be seen as an instance of the famous Pareto principle, which states that, empirically, 80% of the effects (contribution to the complexity of the system) are accounted for by 20% of their causes (those unessential features).

Under the simplicity through reduction label we can perhaps also place factorization and pooling of resources. However, we prefer to keep those under the simplicity through organization label discussed below. This is just the first example of overlapping aspects of simplicity.

Broadly speaking, simplicity through reduction can be related to the Lean Software Development's “Eliminate waste” principle, with waste being defined as something that brings no value to users.

Finally, in reference to the mathematical metaphors of the previous chapter, we can argue that simplicity by reduction implies reducing some of the K-complexities of the system. The description of the IS just got shorter and thus becomes easier to understand. It is also likely that some entropy, due to the weak predictability of manual maintenance operations, will disappear when duplicated systems are suppressed.

Simplicity through reduction is in essence a principle of sobriety.

2.2.2. Simplicity by hiding complexity

While reducing complexity by suppressing features or services from a system is an objective transformation, there are other more subjective ways to increase simplicity. Hiding complexity is such a means to create a useful illusion. Simplicity thus has an objective aspect, namely the reduction of complexity as described in the previous section, but it also has a more subjective angle that is by no means less important or less useful.

Hiding complexity is often related to removing constraints for users. Take single sign-on (SSO), for example. What SSO really does is free users from the constraint of logging into several applications.

More generally, computers are really nothing but machines for hiding complexity of computational tasks.

Hiding complexity means putting it far away. What we have in mind here is the SaaS⁶ model for enterprise computing, whose principle is to outsource altogether the infrastructure of commodity applications to datacenters of services maintained by large service providers. We will have more to say about the SaaS model in section 4.1.2.

By contrast with Simplicity through reduction, this new form of simplicity usually has a cost in complexity that should be carefully weighed. Both the descriptional complexity (K-complexity) and the organized complexity (Bennett's logical depth) are likely to increase overall when complexity is being hidden. The description of the system will be longer and hiding its complexity requires more design input. To put it simply, hiding complexity will promote the creation of value but will also require more design; these costs and benefits should be carefully weighed.

The pertinent question to ask is: When is it really worthwhile to invest in more design to produce more simplicity? One answer is that the encapsulation should benefit to sufficiently many and for a sufficiently long time to compensate for the additional design effort that hiding complexity requires. We shall give another element of response once we have defined what we mean by Simplicity through learning, which shares a deep connection with Simplicity through hiding complexity.

For the moment, to better understand what the latter means, let us pick some examples for the three categories of users of an IS.

2.2.2.1. Customers

Hiding complexity from customers is vital for a company that sells or advertises goods or services through an online platform. To encourage customers to return, everything should be configured for them to navigate a site comfortably. Special attention must be paid to ensuring that all relevant information are easily accessible. The slightest difficulty in finding information might result in customers massively leaving for the competition. This is of course the fascinating subject of graphical user interface usability, a topic that would deserve its own book and which corresponds to the expertise of graphical designers.

2.2.2.2. Business analysts

To make sound decisions, business analysts need, in principle, to run complex queries on one or more databases. Most of them, however, have had no training with the SQL (Structured Query Language) language. Reporting tools and business-rule management systems serve to hide most of the technical complexities involved in writing such queries. Often, they provide some sort of graphical mechanism as a way to build moderately sophisticated business queries.

2.2.2.3. IT personnel

Hiding complexity takes on different forms for people in charge of maintaining an IS. For developers, hiding complexity corresponds to the software design pattern known as encapsulation of data in object-oriented programming (OOP). Encapsulation in this context means that data belonging to a class (in the form of private attributes) shall never be manipulated directly. The only way to manipulate data should be through operations (methods) that guarantee the coherence of that data. The complexity of maintaining the coherence of data is being hidden by those operations. Operations can then be used blindly by other parts of the code, which will thus be exempted from handling data inconsistencies. The concept of interface in OOP is nothing but a very precise specification of how complexity should be hidden for specific purposes.

More generally, hiding complexity is related to the concept of service. The concept of service is a rather vague one because it occurs at so many different levels of IS architecture. As we just described, a service can just be a piece of code, typically a class, which provides some functionality to other parts of the code. More loosely speaking, however, a service can also be any form of encapsulation of an existing functionality, which should be made more broadly available. A common solution is publishing services in an enterprise directory where they can be looked up.

There is yet another, more advanced, form of encapsulation in software engineering, which is represented by frameworks. Because it is closely related to simplicity through learning, we postpone our discussion of this subject until that relevant section.

2.2.3. Simplicity through organization

Organizing is most easily described as performing any kind of action that helps combat disorder. Disorder is to be understood here as the unstructured state. Roughly speaking, organizing a system thus means putting structure into something that initially does not have much.

In an IT context, there are a number of well-known factors that contribute to increasing the disorder in an IS until it becomes literally unmanageable.

– Accumulation of quick-and-dirty pieces of code.

– Diversity of technologies that need to be linked together.

– Progressive obsolescence of technologies.

– Progressive memory loss regarding what has been created, due to employee turnover.

– Absence of an overall modeling of the IS.

– Finally, the chaotic forces of politics often play a significant and even dominant role.

A thorough discussion of sources of complexity in IT will be the topic of Chapter 4.

At a more fundamental level, however, the most remarkable feature of disorder is that it tends to increase spontaneously, even without any apparent cause. This affirmation seems like a joke but in fact has firm foundations in physics. It was formalized during the 19th Century in the famous second principle of thermodynamics⁷. Thermodynamics prescribes limitations on how energy flows can or cannot occur between systems but does not provide an explanation for why the entropy increases. However, to understand how to fight disorder, we really need to understand the mechanism that lies behind this spontaneous increase of disorder. Despite the fact that this is a deep subject of theoretical physics⁸, its essence can be grasped intuitively. Think of a complex system as being composed of a number of subsystems or parts. Statistically, there are overwhelmingly more unstructured configurations than structured configurations. In the context of IT, such a system could be a piece of code, for instance, and “being structured” would simply mean that the code does what it is meant to do.

Briefly referring to section 2.1, let us recall that the Shannon's entropy was a measure of the amount of randomness present in a system. Let us denote the number of configurations available to a system by N. Assume for simplicity's sake that each of these is equally likely to occur: then, as is shown in Appendix 1, the entropy reduces to log N. Hence, the more configurations available to our system, the higher its disorder will be, thus confirming our intuition.

It can also mean a slightly different thing, namely decreasing the number of possible configurations in the description of the system rather than in the system itself. How can we achieve this? Well, mostly by grouping or reusing various parts, either in the physical world of systems or, conceptually, in the realm of models and ideas. This question of reuse will be investigated, from a practical IT point of view, in the subsection “implementing simplicity” in section 4.2.

Prioritizing tasks is a form of organization as well, but one that concerns transformation of processes rather than transforming systems. Essential things first!

Organizational processes are typically slow because they require insight and careful thinking. There is unfortunately no science of organization, but recall from the previous chapter that Bennett's logical depth was precisely the mathematical metaphor for those forms of complexity that can only grow slowly.

Experience shows that organizing a system is an iterative process. When finished, it should result in a system whose description has fewer components than the original system. The system becomes more predictable and easier to understand as a whole, making it easier to modify if necessary. The ability to understand the system as a whole is essential and relates to the next form of complexity, which is discussed in section 2.2.4.

Finally, note that decreasing the randomness of software is also one of the aims of Lean Software Development, according to the motto “Decide as late as possible”. Delaying implementation decisions as much as possible will results in decisions being based on more facts than speculations.

2.2.4. Simplicity through learning

Knowledge and understanding make complex things look simpler. They are both achieved through learning, the process by which the human mind creates an efficient mental map that applies to a category of problems. This mind map is indeed conceptually similar to the concept of a “good” model that we discussed in section 2.1.4: if the mind map is to be useful at all, it should not be too deep. In other words, this mind map should result from a substantial learning effort to be readily useful in a practical situation.

Among all the facets of simplicity we have touched on so far, this is probably the most intricate facet because it relates to nearly every other facet. When applied in an IT context, this is really where the entanglement between the technical and human aspects of an IS come into play.

We shall actually distinguish two aspects to the learning facet of simplicity. We claim that learning can either obviate the need to hide complexity or can help transform complexity from one form into another.

2.2.4.1. Learning obviates the need to hide complexity

The point here is most easily made using examples taken from IT: learning frameworks and learning some simple syntax.

1) The first example is taken from software architecture and concerns frameworks. Put simply, frameworks are nothing but pieces of software architecture that solve well-identified and recurrent design problems that should not be tackled anew on each project. An object-relational framework, for instance, helps construct the object-relational layer of a software architecture, which relates a database to clusters of business objects. Similarly, an injection-dependency framework implements an aggregation mechanism, which guarantees that software components will remain independent of each other, thus ensuring future flexibility. In essence, each framework is nothing but an advanced means to hide algorithmic complexity. The amount of complexity hidden in such frameworks is often quite large. It is not uncommon that it represents tens of thousands of lines of code⁹. The reason behind this complexity is that the range of problems a typical framework intends to solve is very broad.

It is very important to realize, though, that hiding all this algorithmic depth does not come for free. To be parameterized properly, frameworks require a lot of specialized knowledge and experience, which is by no way intuitive and easy to learn. Moreover, frameworks often introduce a set of abstractions, which take some time to become familiar. The point we are trying to make here is that there are numerous situations in IT where using frameworks really represents a form of overkill, all the more so when the problems at hand are reasonably simple.

The message is thus simple: even in 2012, learning how to make explicit, plain-old SQL queries and how to manage transactions “by hand” is still useful and valuable; in addition, learning how to convert data from tables into graphs of business objects will also be useful and valuable.

We shall come back more thoroughly to these issues later when we discuss abstraction in section 2.3.2.

2) The second example we have in mind to illustrate how learning can avoid the need to hide a lot of complexity concerns graphical tools. When designed properly, these are intuitive and user-friendly; hence, it is accepted by users. Since they require no prior technical knowledge, such graphical tools are really just sophisticated “complexity hiders”.

Take Gmail, Google's well-known mail service. It offers very powerful search features. The simpler part of this is available in the usual way: through a standard form that the user has to complete. For the most advanced features, however, the user is kindly asked to learn a little syntax. This small investment in learning some elementary syntax will be rewarded by the time saved when creating such complicated queries as: get all mail between dates 1 and 2 that had no attachments but were starred items.

More generally, experience shows that designing graphical tools is often very costly. They often hide unsuspected quantities of technical intricacies in the generation of the graphics and in the various surface checks they imply. But there is good news: in many situations, this complexity can be handled quite efficiently by human minds instead! There is nothing wrong with graphical tools, we all like them, but the decision to build them should be carefully weighed against the more rustic alternative of having humans learn a little technicality.

We summarize the above discussion in the motto of sobriety below. We believe it corresponds to an essential aspect of simplicity even if it is often overlooked because of an exaggerated, and somewhat naive, faith in the possibilities of technology:

Computers are good at hiding complexity. Humans are good at learning complexity. Systematically relying on complexity hiding is a weak and bad idea. Human brains are still extremely competitive in fighting complexity by learning.

2.2.4.2. Learning allows complexity transformation

Coming back to the example of frameworks mentioned earlier, it should be acknowledged that there are many situations for which their use is fully justified. Such is the case when they are used for solving many complex issues that would otherwise require an unrealistic amount of expertise to be coded explicitly. But, again, this encapsulation does not come for free. The deep algorithmic complexity is partly converted into complex settings in the framework's configuration files and into a set of abstractions that will require some effort to learn.

Using the mathematical metaphors of the previous chapter, we could say that some Bennett logical depth (specialized expert knowledge) has been traded for more K-complexity (long XML configuration files) and higher abstraction.

We summarize the above as:

One of the Lean Software Development principles, namely “Amplify learning”, also touches the learning process. Continuous learning is considered an essential feature of the development process. Learning concerns technical aspects. But equally important is how individuals get to know each other and to work together efficiently and form a team. Learning is also required from end-users who should “learn” what their precise needs are by trying out the successive product releases that are the deliverables of the iterations. To make this learning process efficient, another Lean Software Development principle: “Deliver as fast as possible”, emphasizes that time between two successive iterations should be short. This promotes early feedback from users and better communication within the team.

This final remark regarding speed brings us naturally to the next form of simplicity.

2.2.5. Simplicity implies time saving

Whenever an interaction with an object, a system, an individual, or an organization helps us save time, we feel a sense of simplicity. Time saved is that much more time we can use for something else. Ultimately, it means more freedom. As with complexity, there are both objective and subjective aspects to this form of simplicity. On the objective side, a search engine returning a list of answers quickly provides a sense of simplicity. This is obviously related to performance and efficiency of IS.

Sometimes, however, the same sense of simplicity can be achieved by merely displaying a progress bar that indicates the remaining time in a given process. This subjective shortening of time belongs to ergonomics and we will not have much more to say on this.

2.2.5.1. Lack of time

Before going further, lets us note that time is linked to simplicity in another way. That is, lack of time translates, sooner or later, into lack of organization at some level of the architecture. This lack of organization, in turn, translates into an increased unpredictability:

The most extreme form of unpredictability is of course chaos. Incidentally, the link between speed and chaos has actually been theorized by some essayists [VIR 09], who prophesize that our civilization is facing a “global accident” in the near future. Restricting this analysis to an IT context, the global accident is to be interpreted, more modestly but more realistically, as the collapse of the IS. These things do indeed happen.

2.2.5.2. How simplicity saves time

Now, let us look how the various forms of simplicity we have considered so far can contribute to saving time.

Reducing the number of duplicated systems implies a lower maintenance effort. Manual synchronization procedures will disappear, too. The system is likely to act in a more predictable way, thus saving time spent debugging paranormal-looking phenomena. Suppressing non-essential features from an IS allows and sometimes even constrains users to focus on the basics. As a consequence, these basics will be quicker to master.

Hiding complexity obviously saves a lot of time to those who benefit. Let us recall our examples: graphical tools and frameworks. To put things simply, hiding complexity provides saving in learning time. Ergonomics in a way is nothing but the art of instilling a sense of immediate familiarity that will exempt users from explicit learning.

Organizing complexity reduces the number of subparts of a system, which thus becomes more intelligible to the human mind and more predictable. The learning effort required to have a reasonable overview of the system as a whole is thus strongly reduced. When the need for changes arises, these will be easier and faster to implement.

Learning, at first sight, might seem contradictory with saving time. In fact, learning is better considered a form of time investment which, in the end, will provide some time saving.

2.2.6. Simplicity needs trust

We can either trust in people or trust in systems. What is the relation here to simplicity? As far as trusting in people is concerned, things are more easily analyzed when we look at the consequences of a lack of trust. Can we trust that people will not empty our bank account? Can we trust that people will not use our private communications for their own purpose? Can we trust that people will declare their proper identity when logging in to a corporate site? Sadly, the answer to all these questions is: of course not! The technical consequences of this gave rise to the subject of IT security and cryptography, which is notoriously one of the most difficult (and very fascinating) in IT.

Lack of trust implies designing many verification mechanisms:

– Checking a user's access rights, namely that he or she is granted the appropriate rights to access a resource.

– Identifying and authenticating a user implies checking that the user really is who he or she claims to be.

– Non-repudiation is ensuring that a user cannot claim to have not received a message that was actually received.

– Checking that a private message was neither intercepted nor modified goes under, respectively, privacy and integrity.

– Checking the signature of a message is asserting that it emanates from an identified source.

The above remark can be generalized in at least two directions. First, on the negative side, as the behavior of users cannot be reliably predicted, either because we doubt their competence or their moral integrity or both, additional checking mechanisms should be implemented. On the positive side this time, once we can trust someone, this simplifies his or her interaction both with systems and with other humans. The practical implications of this are not for the IS, which should be designed for average humans (those not trustworthy), but rather for the organization of teamwork and how to grant responsibilities. An organization can strongly benefit from trusted individuals.

Trusting systems reduces to reliability, a form of predictability that follows from simplicity through reduction, hiding, and organizing.

The Lean Software Development principle named “Empower the team” implies, among other things, that people should be trustworthy because this will favor reliable communication within the development team. This is sometimes summarized in the following aphorism:

As a visual summary, there are six simplicity principles that we will refer to later.

2.2.7. What does software architecture tell us?

Information theory has provided us with three deep metaphors of complexity. The laws of simplicity, on the other hand, as proposed by designer John Maeda, inspired six facets of simplicity. Introducing complexity through these non-IT perspectives had the advantage of providing us with a deeper understanding of the nature of complexity and simplicity. Rushing into IT specifics would never have allowed such a broad view. However, as was particularly apparent from our trip into information theory, there is a price to pay for this depth and generality, namely the non-computability of complexity. By contrast, jumping now to the much narrower topic of software architecture will provide us with examples of computable complexity.

Closely examining software architecture is motivated not only by the examples of computable complexity it provides, but, more importantly, by the need to mitigate the source of complexity that it represents within an IS.

Section 2.3.1 examines two examples of computable complexity that apply to code and the IS architecture as a whole. Then, in section 2.3.2, we discuss the role of abstraction in code, how it is either part of the problem of generating needless complexity or, contrariwise, when it actually promotes simplicity.

2.2.7.1. The complexity of code and of IS architecture

The technical architecture of an IS can be considered a generalization of software architecture. Therefore, understanding what complexity means for the latter will help understand better what it means for the former. Our first example, cyclomatic complexity, will be specifically about the complexity of a piece of code of a function or software module. The second example, scale-invariant complexity, applies to general, multi-scale software architectures.

The literature on code complexity is actually quite impressive, if not for its quality, at least for its extensiveness¹⁰ [DUG 05]. Any reasonably thorough review of the subject could easily require a book of its own. The two simple concepts we have picked out, cyclomatic complexity and scale-invariant complexity, have the advantage of being easy to compute practically. Moreover, they both illustrate the price to pay for concreteness, namely some arbitrariness in their definition. Investigating the scale-invariant metric, we also give ourselves the opportunity to propose a practical definition for the concept of scale we introduced earlier. What both examples have in common is that they perform a combinatorial analysis of the information flow within a set of components.

To streamline reading, the more technical details are deferred to Appendix 2.

2.2.7.1.1. Cyclomatic complexity

Among the myriad metrics of code complexity available, cyclomatic complexity is probably the best known. Most code analysis tools allow its measurement. In short:

Cyclomatic complexity is best defined by associating a control-flow graph to the code under consideration. An example is given below:

This control-flow graph represents all possible execution paths through module. Cyclomatic complexity C_cycl is defined as follows:

This perhaps abstract-looking definition of C_cycl fortunately turns out to be very easy to compute. If Eand N denote, respectively, the number of edges and nodes in the control-flow graph, then it can be shown that

Many books actually take the above as the actual definition, but this really obscures the true combinatorial significance of the cyclomatic complexity in terms of independent paths.

Practical wisdom has suggested that a module should not have a cyclomatic complexity much larger than 10. There is substantial evidence that beyond that limit the code becomes just too complicated to be maintained reliably.

2.2.7.1.2. Scale-invariant complexity

Recall that one of the main conclusions from our quick tour of information theory was that complexity, regardless of how it is defined, requires the specification of a scale of description of the system under consideration. ISs, obviously, have many scales of definition that, roughly speaking, correspond to the various architecture levels traditionally referred to in IT: business-process architectures, application architecture, software architecture, physical-deployment architecture, and so on. The question then naturally arises: “Faced with a multi-scale description of an IS, is it possible at all to define a measure of complexity for IS that would be both computable in a reasonable time and scale-invariant?” Even though it is by no means obvious to us that scale invariance is truly desirable, the question, at least, makes sense. The short answer to this question is: yes. We now briefly present such a complexity measure as an example of the arbitrariness involved in defining computable complexities. It was proposed by Yves Caseau et al. [CAS 07].

To define a scale-invariant measure more formally, the first step is to define a mathematically practical abstraction of a multi-scale architecture. Recursive graphs do just this. They can be thought of as a set of nested graphs at different scales. Rather than giving a formal definition, we simply describe it using the figure below. This is discussed in more detail in Appendix 2.

As a practical illustration of the above definitions we propose the following:

– G₁ could represent a high-level SOA overview of an IS considered as a set of services.

– G₂ could represent how services coordinate existing applications to do their work.

– G₃ could represent the software modules that comprise each application.

To discuss scale invariance, we also need a formal concept of the zoom-in and zoom-out operations on a recursive graph. When we zoom out, the low-level, small-scale details become blurred. When we zoom in, small-scale details reappear. This too can be formalized. Technical restrictions apply, however, to the admissible set of recursive graphs.

It is then assumed that each node of the recursive graph is characterized by a unit cost of complexity. These weights must meet a simple compatibility condition: the weight of a node at a given scale should be the sum of the weights of the (smaller-scale) nodes that it contains. It is by no means obvious that such a condition actually holds in real IT systems. However, this condition is essential for the definition below to make sense as a scale-invariant measure.

Let ν_j denote the j^-th node in the smallest scale graph within the multi-scale graph. Let (ν₁…ν_p) be a path of length p (i.e. an order sequence with possible repetitions of length p) on these nodes. Let w_n(ν_j) denote the weight of the node ν_j. The explicit formula for the scale-invariant measure of complexity is then defined using a sum over such paths of length p :

The integer p is arbitrary, each value giving a different definition. The actual proof that this expression is really scale-invariant under the above zoom-in and zoom-out operations relies on elementary combinatorial analysis, detailed in the Appendix 1.

To make the above definition slightly less cryptic and to gain some insight, let us look at two extreme examples.

2.2.7.1.2.1. The spaghetti graph

As a simple metaphor of a maximally messy IT architecture we consider a graph where n nodes are connected with all other nodes. Set any p and take equal weights w_i = 1 on all nodes for simplicity. We must sum over paths with repeated vertices; each node v in the path (ν₁,…,ν_p) can be chosen independently. There are n^p of them, from which we conclude that

2.2.7.1.2.2. The hierarchical graph

As a simple metaphor of a more organized IT architecture consider G_hierachical as mentioned above. Once a starting node ν1 has been chosen (there are n possibilities) each node ν in the path (ν₁,…,ν_p) can be chosen in at most four different ways, we now have

Take n = 100 and p = 2 in the above formulas as an example. The complexity of the spaghetti graph is 100, while the complexity of the hierarchical graph is less than . This is actually true for any large number of nodes n and accounts for the intuitive fact that spaghetti architectures are messier than hierarchical architectures.

The lesson to be learned here is that there is a (high) price to pay for computability: artificial constraints have to be imposed on the structure of the admissible graphs and on the structure of the weight functions. This is to be contrasted with the deep and intrinsic definitions of complexity given in section 2.1.

2.2.8. Abstraction in software engineering

2.2.8.1. Abstraction is everywhere in software

Abstraction is at the core of software engineering. The reason for this is easy to understand: on a human scale, the tasks that software performs are best described in natural language. At the other end, however, processors have to deal with binary sequences. To fill this gap between the machine world and human scale, a range of abstractions, languages, and notations have been designed over the ages: assembly language, structured programming languages, OOP, APIs, patterns, modeling languages such as UML or BPMN, and so on. Computer specialists thus encounter many levels of abstraction daily, whether they are aware of it or not and whether they like it or not! As it happens, people hold strong and opposing views regarding abstraction. Some venerate it, while others abhor it. We shall come back to these more psychological aspects at the end of the following section.

2.2.8.2. Depth and scale revisited

In section 2.1.4, we concluded that the abstraction level of a model (of, e.g. an IS) is essentially defined by a scale of description of a system and a level of compression of information. Let us have a closer look at how both are interdependent. The scale of description is intuitively related to the amount and the nature of the information that is left out from a description or a model. However, information can be left out for very different reasons:

– The information is irrelevant to the current level of description.

For instance, when modeling business objects in an analysis UML class diagram, the programming language is of no relevance. Any information related to language specifics will (and should!) be left out.

– The information is relevant to the current level of description but is assumed to be otherwise easily accessible.

For instance, when modeling business objects in a design UML class diagram, the precise Java type of each attribute might not be explicitly specified because rules have been agreed upon on how to map generic UML types to Java types.

In information theory language: the decompression to perform is moderate because the model is shallow.

– The information is relevant to the current level of description but is not yet explicitly known. Some information is still to be found or created or designed.

Consider again an analysis UML class diagram, modeling business objects. At the time of modeling, the exact cardinalities at each end of the associations might not be known, even if they will be very relevant in the end.

In information theory language: the decompression to perform is important because the model is deep.

It is quite natural to try classifying the various abstraction levels, going from the less abstract to the more abstract. As it happens, this is less obvious than it looks. Consider, for instance, the three following abstraction levels:

1) UML diagrams used while defining requirements of an application (commonly known as the analysis phase). These diagrams usually formalize the functional requirements using a set of diagrams such as use-case diagrams and business-object diagrams and identify perhaps a few services.

2) UML diagrams used for designing the IS, on the other hand, are typically much more detailed and will also include details on the associations between business objects and signatures of methods in each service.

3) The code used in the programming language to build the application. By definition, code is explicit and leaves out no information.

If we decide a language is more abstract the more it leaves out a large amount of information we would conclude that:

[2.1]

This is typically how abstraction levels are organized in UML textbooks. Suppose now we would like to classify the natural language, for example, when used to describe business requirements, according to the above criterion. Natural language embeds much implicit information when compared to explicit computer code. According to our previous rule we should then write:

[2.2]

This, however, looks strange and counterintuitive. Indeed, common sense correlates abstraction of a language or of a notation with a sense of remoteness from our daily intuition. It seems our intuition thus combines scale and compression in a non-obvious way to produce a sense of remoteness. The human mind has different abilities to decompress various kinds of information. It is much more efficient to decompress information from natural language than from UML analysis diagrams, which explains the apparent contradiction between [2.1] and [2.2].

This is not a problem, however. It is rather a hint that:

We will come back to this shortly.

2.2.8.2.1. Object-oriented programming

There is no doubt that OOP is currently the most common paradigm in programming. It introduces a set of abstractions and concepts that are well known to most programmers (classes, interface, heritage, polymorphism, etc.). Let us recall here that the initial goal of OOP was to provide a way to build large software applications by aggregation of smaller modules, while previous approaches, like structured programming, emphasized the opposite approach, namely decomposing large software into smaller modules.

The set of abstractions that OOP introduces is rather small and quickly mastered. It is grossly inadequate, however, to address most problems posed by enterprise computing. For this reason, the original OOP set of abstractions has been consistently extended following a now-standard scenario:

1) A set of basic abstractions on top of OOP, known as APIs, are defined to wrap the most common computing tasks such as building graphical user interfaces, connecting to databases or to directories, building persistence mechanisms, handling strings, reading and writing files, and so on.

2) The above APIs are often so complex that they require real expertise to be put to good use. This expert knowledge is then translated into a catalogue of software patterns. A pattern is most easily defined as a documented formalization of one best practice in software engineering, specialized to OOP¹¹. The UML notation is used most often for this.

3) But software patterns are not yet software! There is still a long way to go from patterns to their actual implementation. This is where frameworks come into play: they supply reference implementations of some particularly important patterns. Currently, there are frameworks for patterns such as the MVC¹², IoC¹³, or DAO¹⁴. As we discussed in section 2.2.4, they encapsulate expert knowledge that is being traded for some significant amount of parameterization and abstraction.

4) Now, even with frameworks available, writing robust code is still not an easy task. One approach that has been recently advocated is model-driven architecture (MDA) where explicit coding is replaced by designing UML diagrams. These will be converted into explicit code for specific platforms and specific frameworks.

This stack of technology, tools, and abstractions is by no means trivial to master. It requires quite a substantial effort, intellectual autonomy, and conceptualization abilities to be properly mastered.

This is the central question that we tackle in the next section.

2.2.8.2.2. Good or bad abstraction?

When used properly, the introduction of a set of abstractions in software engineering can significantly contribute to mitigating many of the recurring problems in IS:

– Improving code quality. The encapsulation of expert knowledge in frameworks, using a set of well-conceived abstractions, promotes code quality, especially when used by beginners who otherwise would have to write, test, and debug such code from scratch. In principle, the uniformity of the code will be increased, thus maintainability will be improved as well, because many developers will have some knowledge of the most current frameworks.

– Promoting the openness of the IS. Opening the IS to other applications or other ISs can be achieved in basically two ways. First, web services can be made available that wrap part of the existing functionality of the IS to make it available to the external world. Web services are most easily described through WSDL¹⁵ documents that are an abstract, technology-independent description of their functionality. Next, APIs are another way to open a system to provide tight integration of an external application into existing code. In a way, APIs can also be considered a network of contracts (interfaces) that precisely describe how to interact with existing code. In short:

– Protection against the fast pace of technological evolution. The idea of a contract, which is a particular form of abstraction, is also an essential way to allow a technology or an algorithm to evolve and improve, while keeping existing functionality stable. The set of interfaces make up the stable part of architecture, allowing the underlying implementation to progressively improve in reliability and performance.

The above arguments are the most often-quoted reasons for promoting the use of frameworks. Other approaches, such as MDA, promote “raising” abstraction by using platform-independent models and diagrams, the so-called PIMs. The idea then consists in building IT systems through modeling rather than the traditional way, through coding. Models and diagrams are indeed expected to be more natural to the human mind than computer code.

Confronting the above arguments with real-life projects shows that things are not really that obvious. It is by no means true, for instance, that more abstraction is systematically the way to go. Quite on the contrary, there are serious drawbacks to excessive or misused abstraction that cannot be overlooked. We now briefly discuss them.

– Multiplication of abstraction layers can needlessly complicate and obscure the code for simple tasks. Frameworks are usually created to solve intricate technical issues. One example is solving O/R mapping problems in a transactional and distributed setting. Another is structuring code in a modular way using the IoC pattern. Most often, frameworks also include security features and these bring their own complexity. There are many circumstances, however, where such tools are overly complicated and could profitably be replaced with more explicit and basic coding that bypasses the whole framework stack.

– Relying too much on frameworks and their set of abstractions can promote the dangerous illusion, for developers, that many of the basic programming skills, such as an operational knowledge of the basic patterns, have now become obsolete. The framework will take care of everything. In the worst case, the original rationale of a framework will be forgotten by developers. They will use it … just because it is there and it is the tradition to use it! In the end, the consequence of such blind and indiscriminate use of frameworks’ abstractions will be a progressive loss of basic technical skills. This matter of fact exacerbated by the belief that computational resources are for all practical purposes limitless. Thus, the basic idea that the complexity of a solution should be reasonably proportional to that of the problem it addresses is progressively lost and, along with it, a healthy intuition regarding the performance of ISs. We will come back to this in section 4.3.2.

– As we have already discussed in section 2.2.4, frameworks really provide a special form of complexity hiding or, more accurately, a form of complexity conversion:

As is implicit in this statement, the conversion does not come for free. Some learning effort is needed before the complexity encapsulated in a framework becomes relevant, in the sense that it provides effective time saving. This is just a restatement of the principle of simplicity through learning. It is thus essential that this learning period, which could span several weeks or months for some complex frameworks, be compared with other relevant timescales:

1) The first timescale is related to the sustainability of the framework itself. Modern, popular frameworks are now most often developed by the open-source community. They are quite stable, well-documented, and have a lifetime that largely exceeds the learning time. There is thus no real worry about their sustainability. By contrast, many frameworks have been and still are developed in-house. These often will not benefit from such sustainability, as they are often related to eccentric, temporary, and local technological endeavors. They are currently a major cause of needless complexity in ISs.

2) The second timescale to consider is related to the period during which this framework will be used, once it has been reasonably mastered, either by an individual developer or by an IT team. In other words, the rate of reuse is an important parameter to consider as well. For the learning investment to be reasonably profitable, the framework in question should be used more than once. As a rule of thumb, we can consider that using a technology or a set of abstractions only once is a waste of time and energy.

As we can see, there is thus no quick and easy answer as to the usefulness of a set of abstractions. But three timescales should to be kept in mind and compared:

– The time needed to fully grasp a set of abstractions.

– The sustainability of a set of abstractions, which determines for how long they will be around.

– The effective rate of reuse once they have been learned.

These should all be lucidly balanced before making a decision to add any abstraction layer to the software architecture, if needless complexity is to be avoided. Such a sound decision is, however, often made quite difficult when most IT skills come from external software engineering companies. Developers hop from one project to another and face such a large number of technologies that they cannot afford to really master any of them in depth. Most often, they just adapt to an existing technical ecosystem of which they have no clear overview. They end up blindly complying with some local development rules or habits. Often, these were not really thought through, but rather emerged in a specific context. For various reasons (technology hype and local politics) those rules, in turn, often change too quickly to become part of a genuine technical culture which, alone, could turn them into efficient tools.

We shall come back to these important issues in section 4.3, when we discuss human factors to uncontrolled complexity.

2.2.8.2.3. Good abstraction is a form of simplicity!

API and frameworks respectively define and implement sets of abstractions (design patterns). “Good” abstraction is really a mix of various forms of simplicity principles as defined in section 2.2.

– “Good” abstraction hides algorithmic complexity. Frameworks are nothing but an advanced form of hiding expert knowledge on software design.

– “Good” abstraction transforms complexity through learning. Learning abstraction allows transformation of expert knowledge into a set of abstractions and some amount of parameterization.

– “Good” abstraction saves time when designing software. Provided the above balance between timescales is positive, a set of abstractions will indeed save time.

Abstract thinking also plays an important role in IT, even before actual coding can begin, namely during the most delicate period of an IT project, analysis, which consists of transforming a set of user requirements into a design model that can, in turn, be unambiguously transformed into code. Referring to the definition we gave in “Depth and Scale Revisited” for an abstraction level, we can identify at least three levels of abstraction to be used in the delicate process of formalizing user requirements. We assume for simplicity's sake that the modeling is performed using the UML notation.

1) The level we call “Non-formalized what”. This level corresponds simply to a set of requirements as they can be expressed by expert users, that is, in natural language and with no formalization whatsoever. These still contain ambiguities and implicit assumptions that should be clarified in later stages. A set of requirements is usually obtained as a summary of several interviews of expert users. This non-formalized level can hardly be avoided because it cannot reasonably be assumed that all users will master UML or an equivalent modeling language.

2) The level we call “Formalized what”. This is where the formalization of requirements comes in. It is traditionally called the analysis stage. All ambiguities as to what exactly the system should be doing must be raised. The set of UML diagrams that are useful at this stage are typically Use Case and Activity Diagrams that help define a logical partition of the set of requirements, Class Diagrams that define an appropriate set of business objects and a set of business services that will make up the system. This level of description also typically includes a detailed description of the user interface as well as the navigation logic between screens or windows. This level contains no technical information whatsoever.

3) The level we call “Formalized how”. This is the level of description of a system that is appropriate for a software architect. It specifies how the system will be realized, which architecture choices are made, and which frameworks and APIs will be used. Detailed Class and Activity Diagrams are typically used at this design stage. This level contains all technical information left out from the analysis stage. This level may also include information regarding physical deployment: which software runs on which node within a cluster of machines.

4) Explicit code. This is the level we discussed previously. It is itself subdivided into a range of abstractions that belong to the various APIs and frameworks that will be used.

Much more could be said, but we will not go into such detail here because that would be the topic for a book on UML modeling. The main point that we would like to make is directly related to abstraction:

Let us explain this briefly, assuming UML is being used. The issue is that a notation such as UML, although it specifies as many as 13 different types of diagrams, does not specify clear and universally accepted abstraction levels. As a consequence, there is often much ambiguity left as to what, for instance, a class diagram really means. A business expert might expect it to provide a detailed glossary of business objects, whereas a developer will instead expect to find detailed information regarding the type of the attributes and the associations that should exist between classes being coded.

Experience shows that these kinds of ambiguities are a source of much incomprehension, especially among inexperienced IT teams. There are only two remedies for this:

– The first is to define very clearly what kind and detail level of information a particular UML diagram should contain at a given abstraction level. It is quite useful also to define how diagrams should be related to each other. How should Use Case Diagrams be related to Activity Diagrams? Should Class Diagrams at the analysis stage contain attributes? If so, should the attributes be typed? When should data be modeled as an attribute of a class or as an association of a class? There is no simple and easy answer to these questions. However, each IT team should define its own and stick to it.

– The second solution is to keep the number of abstraction levels (except explicit code) limited to no more than three. A typical mistake is that technical details and issues creep into the analysis diagrams where they have no place. In sum:

This way, communication between the business experts and the IT teams will be more reliable. Again, achieving such efficient communication between technical and non-technical personnel staff will not usually happen overnight. It typically requires the establishment of an enterprise-wide IT communication culture, with appropriate training of IT teams and business analysts led by individuals that have both good conceptual skills (to master UML and train their colleagues) and, simultaneously, a good knowledge of the business process (to illustrate concepts with relevant examples).

2.2.8.2.3.1. MDA and abstraction

Let us briefly come back to the MDA approach that we referred to earlier. Recall that the purpose of MDA is to build ISs using UML models (the PIM¹⁶) rather than by explicit coding. We can now rephrase this as an attempt to jump from the “Formalized what” level (no. 2) directly to the code level (no. 4) by automatic code generation. Thus, MDA implies, in principle, skipping the “Formalized how” (no. 3) level altogether. We remain extremely skeptical about the possibility of such an endeavor. First, because the MDA hype has often raised false hopes that applications could soon be generated by sketching a few diagrams and then pushing a button to generate code. We think these are not much more than naive daydreams, at least if MDA is to be considered as a general-purpose tool. The second reason for our skepticism is that there are no serious indications that building the so-called PSM¹⁷ model (as a substitute to the level no. 3) is really simpler or more reliable than good old explicit coding.

2.2.8.2.3.2. DSL and abstraction

We conclude this long section on abstraction with a few slightly more speculative ideas that actually pertain more to ongoing R&D than to everyday project life. Most generalpurpose software-engineering practices currently rely on the OOP paradigm. Objects are the single, basic abstraction on which all other abstractions are based: APIs, frameworks, and finally applications. This process implies successive encapsulation of aggregations of ever-more complex sets of objects. One possible explanation for part of the complexity of current IT is perhaps ultimately rooted in this implicit assumption that this basic object abstraction is the best possible foundation of enterprise computing. No doubt objects can be used in all layers of software architecture. They can be used to define such diverse concepts as a business object, a dynamic HTML tag, a transactional scope, a SQL request, a list of security credentials, or a GUI widget. Some of these uses of objects look quite natural to our intuition. Indeed, thinking of enterprise data as clusters of objects is rather natural. Other concepts, on the other hand, when expressed as objects, look rather contrived. Thinking about an SQL request, a list of security credentials or a transactional scope in terms of objects cannot be considered very natural, or very elegant. There is indeed a substantial learning effort (in the sense of section 2.2.4) that is associated with getting used to this “object-oriented way” of looking at things. Perhaps this effort could be avoided, or at least mitigated, provided the single object abstraction was replaced by a set of more specialized abstractions that would be targeted on different computing or modeling tasks. One language/notation could be used to express business objects, another could be used to express security credentials, still another to express nested transactions, and so on.

This is the so-called DSL¹⁸ approach, which is currently still in its infancy. We consider that this approach is, by far, more promising and more lucid than the MDA approach. Many open questions certainly remain. For instance, we must ask how these different abstractions will be related to each other (this is really a generalized kind of O/R mapping problem) and how normalization will proceed, assuming such normalization is useful.

Note that we used the word “elegance” earlier. Maybe this is actually a clue to be taken seriously. Other domains of fundamental or applied sciences, such as mathematics, modern physics, urban planning, or building architecture, have all made substantial progress by considering that elegance and beauty are to be considered seriously, especially when promoting simplicity and efficiency are at stake. Elegance is important in building architecture because we live in buildings. Elegance is important in mathematics because it is an unconstrained exploration of concepts. Computing, and especially enterprise computing, obsessed as it is by deadlines, has promoted short-term thinking and, so far, has overlooked this aspect of things altogether. Nothing indeed could sound weirder than a claim such as “What a beautiful IS!” Perhaps this is wrong.

 

 

1 Claude Elwood Shannon (1916–2001) was an American mathematician, electronic engineer, and cryptographer recognized as “the father of information theory”.

2 Andrey Nikolaevich Kolmogorov (1903–1987) was a Soviet Russian mathematician, who advanced mathematics and logic, including computational complexity.

3 Charles H. Bennett is a Fellow at IBM Research. Bennett's recent work has concentrated on applying quantum physics to information exchange. He is one of the founding fathers of quantum information theory.

4 CRUD is the acronym for Create, Read, Update, and Delete, which are the basic data manipulation operations on any set of data.

5 Agile software development refers to a group of software development methodologies based on iterative and incremental development, where requirements and solutions evolve in parallel.

6 SaaS is the acronym for Software as a Service.

7 The second principle of thermodynamics asserts that there exists a state function known as entropy and that, for an isolated system, this quantity grows until the system reaches its state of thermal equilibrium.

8 It is related to the question of explaining how irreversibility occurs in macro-systems.

9 The core of Hibernate 3.0 contains nearly 70,000 lines of code and over 26,000 lines of unit tests.

10 A summary of metrics for code complexity with an extensive bibliography can be found here: Mesure de complexité et maintenance de code by Philippe Dugerdil, Geneva School of Business Administration. http://lgl.isnetne.ch/isnet72/Phase3/complexiteCode.pdf.

11 The Core J2EE Patterns is such an example. http://java.sun.com/blueprints/corej2eepatterns/Patterns/.

12 MVC stands for Model View Controller pattern, which encourages the separation of user interface logic, business logic, and the data they operate on.

13 IoC stands for Inversion of Control pattern, which helps build software using modules that are strongly independent from one another.

14 DAO stands for Data Access Object pattern, which is a pattern for abstracting data manipulation within any data repository.

15 WSDL = Web Service Description Language.

16 PIM = Platform Independent Model.

17 PSM = Platform Specific Model is a set of metadata that defines how code generation occurs when targeting a specific platform such as Java Enterprise and .NET.

18 DSL = Domain-Specific Language.